{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Chapter 6: Deep learning"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Introducing convolutional networks"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Problem 1 ([link](http://neuralnetworksanddeeplearning.com/chap6.html#problem_393174)): equations of backpropagation in a convolutional network\n",
    "\n",
    "We consider a network like the following, except that **there's only one convolutional layer and only one pooling layer:**\n",
    "\n",
    "![img/conv_network.png](img/conv_network.png)\n",
    "\n",
    "The leftmost arrows actually look like this:\n",
    "\n",
    "![img/conv.png](img/conv.png)\n",
    "\n",
    "And the second arrows like this:\n",
    "\n",
    "![img/max_pooling.png](img/max_pooling.png)\n",
    "\n",
    "The third arrow actually represents a full connection between the max-pooling layer and the output layer.\n",
    "\n",
    "We'll call:\n",
    "\n",
    "* $a_{j,k}^0$, $0 \\leq j, k \\leq 27$, the input activations;\n",
    "* $w_{l,m}^1$, $0 \\leq l, m \\leq 4$, the shared weights for the convolutional layer;\n",
    "* $b^1$, the shared bias for the convolutional layer;\n",
    "* $z_{j,k}^1$, $0 \\leq j, k \\leq 23$, the weighted input to neuron $(j, k)$ (line $j$, column $k$) in the convolutional layer:\n",
    "\n",
    "$$z_{j,k}^1 = b^1 + \\sum_{l=0}^4 \\sum_{m=0}^4  w_{l,m}^1 a_{j+l, k+m}^0$$\n",
    "\n",
    "* $a_{j,k}^1$, $0 \\leq j, k \\leq 23$, the activation of neuron $(j, k)$ in the convolutional layer:\n",
    "\n",
    "$$a_{j,k}^1 = \\sigma \\left(z_{j,k}^1 \\right)$$\n",
    "\n",
    "* $a_{j,k}^2$, $0 \\leq j, k \\leq 11$, the activation of neuron $(j, k)$ in the max-pooling layer:\n",
    "\n",
    "$$a_{j,k}^2 = \\max \\left( a_{2j, 2k}^1, a_{2j, 2k+1}^1, a_{2j+1, 2k}^1, a_{2j+1, 2k+1}^1 \\right)$$\n",
    "\n",
    "So neuron $(j,k)$ in the convolutional layer will contribute to the computation of the max for neuron $\\left( \\left \\lfloor{\\frac j 2}\\right \\rfloor, \\left \\lfloor{\\frac k 2}\\right \\rfloor \\right)$\n",
    "\n",
    "* **Note that the max-pooling layer doesn't have any weights, biases, or weighted inputs!**\n",
    "* $w_{l;j,k}^3$, $0 \\leq j, k \\leq 11, 0 \\leq l \\leq 9$, the weight of the connection between neuron $(j,k)$ in the max-pooling layer and neuron $l$ in the output layer;\n",
    "* $b_l^3$, $0 \\leq l \\leq 9$, the bias of neuron $l$ in the output layer;\n",
    "* $z_l^3$, $0 \\leq l \\leq 9$, the weighted input of neuron $l$ in the output layer:\n",
    "\n",
    "$$z_l^3 = b_l^3 + \\sum\\limits_{0 \\leq j, k \\leq 11} w_{l;j,k}^3 a_{j,k}^2$$\n",
    "\n",
    "* $a_l^3$, $0 \\leq l \\leq 9$, the output activation of neuron $l$ in the output layer:\n",
    "\n",
    "$$a_l^3 = \\sigma \\left( z_l^3 \\right)$$\n",
    "\n",
    "Now for comparison, here are equations BP1 - BP4 for regular fully connected networks:\n",
    "\n",
    "* **BP1**: $\\delta^L_j = \\frac{\\partial C}{\\partial a^L_j} \\sigma'(z^L_j)$\n",
    "* **BP2**: $\\delta^l_j = \\sum_k w^{l+1}_{kj}  \\delta^{l+1}_k \\sigma'(z^l_j)$\n",
    "* **BP3**: $\\frac{\\partial C}{\\partial b^l_j} = \\delta^l_j$\n",
    "* **BP4**: $\\frac{\\partial C}{\\partial w^l_{jk}} = a^{l-1}_k \\delta^l_j$\n",
    "\n",
    "And their shortened derivations (only writing $\\frac{\\partial x}{\\partial y}$ when $y$ has an influence on $x$):\n",
    "\n",
    "* **BP1**:\n",
    "\n",
    "$$\\delta_j^L = \\frac{\\partial C}{\\partial z_j^L} = \\frac{\\partial C}{\\partial a_j^L} \\frac{\\partial a_j^L}{\\partial z_j^L} = \\frac{\\partial C}{\\partial a_j^L} \\sigma'(z_j^L)$$\n",
    "\n",
    "* **BP2**:\n",
    "\n",
    "$$\\delta_j^l = \\frac{\\partial C}{\\partial z_j^l} = \\sum\\limits_k \\frac{\\partial C}{\\partial z_k^{l+1}} \\frac{\\partial z_k^{l+1}}{\\partial a_j^l} \\frac{\\partial a_j^l}{\\partial z_j^l} = \\sum\\limits_k \\delta_k^{l+1} w_{kj}^{l+1} \\sigma'(z_j^l)$$\n",
    "\n",
    "* **BP3**:\n",
    "\n",
    "$$\\frac{\\partial C}{\\partial b_j^l} = \\frac{\\partial C}{\\partial z_j^l} \\frac{\\partial z_j^l}{\\partial b_j^l} = \\delta_j^l \\times 1$$\n",
    "\n",
    "* **BP4**:\n",
    "\n",
    "$$\\frac{\\partial C}{\\partial w_{jk}^l} = \\frac{\\partial C}{\\partial z_j^l} \\frac{\\partial z_j^l}{\\partial w_{jk}^l} = \\delta_j^l a_k^{l-1}$$\n",
    "\n",
    "Let's look at each equation in turn, with our new network architecture.\n",
    "\n",
    "* **BP1**: The last layer following the previous network architecture, we see that the derivation of BP1 remains correct. Therefore, BP1 doesn't change.\n",
    "* **BP2**: since the max-pooling layer doesn't have any weighted inputs, we'll just have to compute $\\delta_{j,k}^1$.\n",
    "\n",
    "\\begin{equation*}\n",
    "    \\begin{aligned}\n",
    "        \\delta_{j,k}^1 &= \\frac{\\partial C}{\\partial z_{j,k}^1} \\\\\n",
    "        &= \\sum\\limits_{l=0}^9 \\frac{\\partial C}{\\partial z_l^3} \\frac{\\partial z_l^3}{\\partial z_{j,k}^1} \\\\\n",
    "        &= \\sum\\limits_{l=0}^9 \\delta_l^3 \\frac{\\partial z_l^3}{\\partial a_{j', k'}^2} \\frac{\\partial a_{j', k'}^2}{\\partial z_{j,k}^1} \\qquad \\text{with } j' = \\left \\lfloor{\\frac j 2}\\right \\rfloor \\text{ and } k' = \\left \\lfloor{\\frac k 2}\\right \\rfloor  \\\\\n",
    "        &\\text{(} a_{j',k'}^2 \\text{ being the only activation in the max-pooling layer affected by } z_{j,k}^1 \\text{)} \\\\\n",
    "        &= \\sum\\limits_{l=0}^9 \\delta_l^3 w_{l;j',k'}^3 \\frac{\\partial a_{j', k'}^2}{\\partial z_{j,k}^1} \\\\\n",
    "        &= \\sum\\limits_{l=0}^9 \\delta_l^3 w_{l;j',k'}^3 \\frac{\\partial a_{j', k'}^2}{\\partial a_{j,k}^1} \\frac{\\partial a_{j,k}^1}{\\partial z_{j,k}^1} \\\\\n",
    "        &= \\sum\\limits_{l=0}^9 \\delta_l^3 w_{l;j',k'}^3 \\frac{\\partial a_{j', k'}^2}{\\partial a_{j,k}^1} \\sigma' \\left( z_{j,k}^1 \\right)\n",
    "    \\end{aligned}\n",
    "\\end{equation*}\n",
    "\n",
    "Now since $a_{j',k'}^2 = \\max \\left( a_{2j', 2k'}^1, a_{2j', 2k'+1}^1, a_{2j'+1, 2k'}^1, a_{2j'+1, 2k'+1}^1 \\right)$ and we're talking about infinitesimal changes, we have:\n",
    "\n",
    "\\begin{eqnarray}\n",
    "   \\frac{\\partial a_{j', k'}^2}{\\partial a_{j,k}^1} = \\left\\{ \n",
    "    \\begin{array}{ll} \n",
    "      0 & \\mbox{if } a_{j,k}^1 \\neq \\max \\left( a_{2j', 2k'}^1, a_{2j', 2k'+1}^1, a_{2j'+1, 2k'}^1, a_{2j'+1, 2k'+1}^1 \\right) \\\\\n",
    "      1 & \\mbox{if } a_{j,k}^1 = \\max \\left( a_{2j', 2k'}^1, a_{2j', 2k'+1}^1, a_{2j'+1, 2k'}^1, a_{2j'+1, 2k'+1}^1 \\right)\n",
    "    \\end{array}\n",
    "  \\right.\n",
    "\\end{eqnarray}\n",
    "\n",
    "This is because $a_{j,k}^1$ only affects $a_{j',k'}^2$ if $a_{j,k}^1$ is the maximum activation in its local pooling field. In this case, we have $a_{j', k'}^2 = a_{j,k}^1$, so $\\frac{\\partial a_{j', k'}^2}{\\partial a_{j,k}^1} = 1$.\n",
    "\n",
    "And so to conclude the derivation of our new BP2:\n",
    "\n",
    "\\begin{eqnarray}\n",
    "   \\delta_{j,k}^1 = \\left\\{ \n",
    "    \\begin{array}{ll} \n",
    "      0 & \\mbox{if } a_{j,k}^1 \\neq \\max \\left( a_{2j', 2k'}^1, a_{2j', 2k'+1}^1, a_{2j'+1, 2k'}^1, a_{2j'+1, 2k'+1}^1 \\right) \\\\\n",
    "      \\sum\\limits_{l=0}^9 \\delta_l^3 w_{l;j',k'}^3 \\sigma' \\left( z_{j,k}^1 \\right) & \\mbox{if } a_{j,k}^1 = \\max \\left( a_{2j', 2k'}^1, a_{2j', 2k'+1}^1, a_{2j'+1, 2k'}^1, a_{2j'+1, 2k'+1}^1 \\right)\n",
    "    \\end{array}\n",
    "  \\right.\n",
    "\\end{eqnarray}\n",
    "\n",
    "* **BP3**: we consider two cases:\n",
    " * $\\frac{\\partial C}{\\partial b_l^3} = \\delta_l^3$ as the third layer respects the previous architecture (the derivation still works);\n",
    " * $\\frac{\\partial C}{\\partial b^1}$. This one is different, since the bias $b^1$ is shared for all neurons in the convolutional layer. We have:\n",
    " \n",
    "\\begin{equation*}\n",
    "    \\begin{aligned}\n",
    "        \\frac{\\partial C}{\\partial b^1} &= \\sum\\limits_{0 \\leq j,k \\leq 23} \\frac{\\partial C}{\\partial z_{j,k}^1} \\frac{\\partial z_{j,k}^1}{\\partial b^1} \\\\\n",
    "        &= \\sum\\limits_{0 \\leq j,k \\leq 23} \\delta_{j,k}^1 \\frac{\\partial z_{j,k}^1}{\\partial b^1} \\\\\n",
    "        &= \\sum\\limits_{0 \\leq j,k \\leq 23} \\delta_{j,k}^1 \\qquad \\text{as } z_{j,k}^1 = b^1 + \\sum_{l=0}^4 \\sum_{m=0}^4  w_{l,m}^1 a_{j+l, k+m}^0\n",
    "    \\end{aligned}\n",
    "\\end{equation*}\n",
    "\n",
    "* **BP4**:\n",
    " * $\\frac{\\partial C}{\\partial w_{l;j,k}^3} = a_{j,k}^2 \\delta_l^3$ since, again, the derivation still works for the third layer;\n",
    " * $\\frac{\\partial C}{\\partial w_{l,m}^1}, 0 \\leq l, m \\leq 4$. These 25 weights are shared, and each of them is used in the computation of the weighted input of each neuron in the convolutional layer:\n",
    "\n",
    "\\begin{equation*}\n",
    "    \\begin{aligned}\n",
    "        \\frac{\\partial C}{\\partial w_{l,m}^1} &= \\sum\\limits_{0 \\leq j, k \\leq 23} \\frac{\\partial C}{\\partial z_{j,k}^1} \\frac{\\partial z_{j, k}^1}{\\partial w_{l,m}^1} \\\\\n",
    "        &= \\sum\\limits_{0 \\leq j, k \\leq 23} \\delta_{j,k}^1 \\frac{\\partial z_{j, k}^1}{\\partial w_{l,m}^1} \\\\\n",
    "        &= \\sum\\limits_{0 \\leq j, k \\leq 23} \\delta_{j,k}^1 a_{j+l,k+m}^0 \\qquad \\text{as } z_{j,k}^1 = b^1 + \\sum_{l=0}^4 \\sum_{m=0}^4  w_{l,m}^1 a_{j+l, k+m}^0\n",
    "    \\end{aligned}\n",
    "\\end{equation*}"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Convolutional neural networks in practice"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Exercise 1 ([link](http://neuralnetworksanddeeplearning.com/chap6.html#exercise_683491)): importance of the fully-connected layer\n",
    "\n",
    "Because I'm running code with a CPU and training the network is quite long (it would take around 30 minutes to train the network with the fully-connected layer for 60 epochs):\n",
    "* I won't train the network with the fully-connected layer and will instead trust Nielsen's result of a 98.78 percent accuracy;\n",
    "* I'll only train the network once, instead of keeping best-in-3 results.\n",
    "\n",
    "For these reasons, the comparison won't be satisfying. To get a satisfying comparison, simply train 3 networks and keep the best result.\n",
    "\n",
    "The code is in the `chap6ex1` directory.\n",
    "\n",
    "`exec_with.py` trains a network with the fully-connected layer (which I haven't done), and `exec_without.py` trains a network without this layer.\n",
    "\n",
    "The best classification accuracy obtained during these 60 epochs is 98.52 percent (at epoch 26).\n",
    "\n",
    "It's worse than Nielsen's accuracy with a fully-connected layer, but I trained only one network, not 3 (as announced, the comparison isn't satisfying).\n",
    "\n",
    "Anyway, the difference seems significant, so the fully-connected layer was probably helpful."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Problem 2 ([link](http://neuralnetworksanddeeplearning.com/chap6.html#problem_834310)): using the tanh activation function\n",
    "\n",
    "In the directory `chap6p2`, `exec_tanh` uses the hyperbolic tangent activation function.\n",
    "\n",
    "To plot accuracies, I've simply modified `chap6p2/network3.py` so that it keeps and prints the list of per-epoch accuracies (variable `past_accuracies`), which I've copy-pasted below."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 2,
   "metadata": {
    "scrolled": false
   },
   "outputs": [
    {
     "data": {
      "image/png": "iVBORw0KGgoAAAANSUhEUgAAAYQAAAD8CAYAAAB3u9PLAAAABHNCSVQICAgIfAhkiAAAAAlwSFlzAAALEgAACxIB0t1+/AAAADl0RVh0U29mdHdhcmUAbWF0cGxvdGxpYiB2ZXJzaW9uIDIuMi4yLCBodHRwOi8vbWF0cGxvdGxpYi5vcmcvhp/UCwAAIABJREFUeJzt3Xt8ldWd7/HPL/crtyQgECRYsYiCURGstqLMaQ/WekFbW0+1Y2/OzZk5naOjtKed1qm1dnzV6TntOONdOha1atX24BVRO696ISh3BFEJhEtIIAm5kMtOfueP50nYSTZk50Ju+/t+vfYre69nPc9eK4Tn96y1nrUec3dERESShroAIiIyPCggiIgIoIAgIiIhBQQREQEUEEREJKSAICIiQJwBwcwWm9lWM9tuZrfG2D7dzFaa2Xoze83MCqO23WlmG8PXl6PSZ5jZ22b2gZk9bmZpA1MlERHpix4DgpklA78CLgZmA9eY2ewu2e4Clrn7XOA24I5w30uAs4BiYAFws5mNCfe5E7jb3WcCVcA3+18dERHpq3haCPOB7e7+kbs3A48Bl3fJMxtYGb5fFbV9NvC6u0fcvR5YByw2MwMWAU+G+R4Bruh7NUREpL9S4sgzFdgV9bmM4Go/2jrgKuAXwBIg18zywvR/MrOfA1nARcBmIA+odvdI1DGnxvpyM7sBuAEgOzv77FmzZsVRZBERabdmzZpKdy/oKV88AcFipHVd7+Im4Jdmdj3wBrAbiLj7S2Z2DvAnoAJ4E4jEecwg0f1e4F6AefPmeUlJSRxFFhGRdmZWGk++eLqMyoBpUZ8LgT3RGdx9j7tf6e5nAt8L02rCn7e7e7G7f5YgEHwAVALjzCzlaMcUEZHBFU9AWA3MDO8KSgO+AjwXncHM8s2s/VhLgQfD9OSw6wgzmwvMBV7yYEW9VcAXw33+HHi2v5UREZG+6zEghP38NwIvAluAJ9x9k5ndZmaXhdkuBLaa2TZgEnB7mJ4K/NHMNhN0+1wbNW5wC/APZradYEzhgQGqk4iI9IGNpOWvNYYgItJ7ZrbG3ef1lE8zlUVEBFBAEBGRkAKCiIgACggiIhJSQBAREUABQUREQgoIIiICKCCIiEhIAUFERAAFBBERCSkgiIgIoIAgIiIhBQQREQHie2KaiIgMgpbWNqoammNum5CVRkry8b2GV0AQETlOGpojlB5ooPRAPXVNrd221xxuofRAPTvCPGVVh2lti/1Iglf+YSEnT8w5ruVVQBCRhNHS2kZZ1WF2HKintDI4ER/tiryv2hzKaxrZcaCe/bVNPebPzUihKC+bOVPHcuncKUwak45Z98fOF+SkD2g5Y1FAkNEl0gTVO6G5/khSm1NR20T2mPGMOeETkHyc/uxbGoPvbmmIL39aDow7EVLSYm52d6obWthTc5h4n2NVkJvOxNzYJ5TB1BRppazqcHD1Wxlc/e6srOVwZSnWWN0tf3pKEieMzWDKuEwmj81kytgMCnLTSUrqXI8kMwpy0snNOPq/YVOkjf21jeypbmRP9WH21jSyp+Yw+2oa2V/b1OkKPDM1iXFZaQz0b2tGdhrnTctg8rhMJo/NYPLYzJhlzkxLZkxGCtZRgsPhK4bUSQQPoTx+FBBkWKutOUj5js3U7N5Ky4FSvC3Sabt5GzlN+xl7eBdjD+8it2kfRuezZwowOXzfSjKRMdNIn3gyTDgJciZBX0+eba1QUwYHP4KDH8Oh3UDvnkDYRhJ1GSdQkzGNmsxCqlIKqDrcSnVDM1UNzTS1tPW6WCnJxvisNMZlpTI+K4301ON/74g71DZGwnK3UNvYgrtTYDXMsH0sStpPoe0nhe7dJgC0AJXhq5/SgWnhq5tY59PG/n9nzGMeGOBj/s1qKDhlgA/amQKCBNraICmOE0dLI1SXhifB8FW1I7gy728R3DtOKk2N9eS37GUCh8jtYb8DnstOn0SJn0Spf4rStkm0pOYycUxwtTwxN52JOelUVu5j/44tTKnay6z6j5m+4y1SI3X9KnNN0ljKk6ewL+WT7Mu9iPLkyTQmZXbK4w51jRH21zURaT0SMMZQz/Sk/Uyv30dRQzkn2iZOty7l6esFYWP4OtjH/fsrPLO0pubg42eQnL8Am3BSEISzJsAxrskbWiJU1DZRVd/SbVukzTlQ18T+2iYqapsor23kYH0zOekpTMxNpyA3g4m5aUFLaUwGk3IzyE5PjroCH8HGTO45Tz8pICQKd2g4cORq9uCHnU/qjYeC7ov2/7QTToKcicFVb0e+j4Mr4uir4PSxMKEo6P7opUhrG42RNuqbIlQfbuHQ4Rba3DEgPSOL7RMW0jZuBumTTmbs1FkUTJtJWnpG54OYkZOcxmxgdlRyWnJSzG6TuqYIv3u3jL98s5Tt+2vJTm6lcHwW0/OyOHFCFtPzshmXmUpZ9WF2Hqin9GADOw80dOsLHpeZyrQJWSSnZXT7jljGjk9lRn420/OyKMoLfubnpHdrnLS2tZCcNHAnr7Y2p6Wt962M3jKMtJTuFxTJyWm9boFlAdPDVzza2rxb15L0jXm8nZPDwLx587ykpGSoizF8uUNdeecTffTJvOnQkbyWBGMLj5z8M8ZCVdSVf3TerLwj+cbPgLxPHPmcOf6o/+HdnYP1zR13UHT9Wd1w5AqwcHwmC08pYOEpBZx3cj456cf3WsXdefPDA7z+QQU7DzR0lKmh+UiXRkFuOkV5WZw4ITyR52dTlJfF9AnZjM06vn25IgPJzNa4+7ye8qmFMJxtfBr++HOIHGWQKZq3Qe2+zgOalhxc9ed9AqYt6HxSHz8dUrrftVDd0Mwft1Wwest2Du7bxdQTP8GC007i3JPyyErr+c8l0trGy5vLefTtnazbVU1t05E+fzOYMjaTGfnZXDJncseV8sxJuRTlZQ3qQKiZcd7J+Zx3cn5HmrtTURd0VRSOzyT7OAclkeFGf/HDUWMNrPhHWP8YTJoDk8+Ib7+cSeFJf0bwc+w0SD72lWxrm7O+rJrXt1Xw+rYK1u2qps1hbGYqnzzhE6xcW8N/rC4hLTmJc2aM54KZBcyaPIaivCymjsvsmChTWdfEY+/s5NG3d7K3ppGp4zJZctZUivKyKcoPumIKx2eSnpLc39/OcWNmTMzNYGJufN1AIqONAsJwU/onePovgr77hbfCBTcP+G2S+w818vq2Ct74oJI/flBBdUMLZnBG4Tj+dtFMFn6ygDMKx5GcZDRFWinZURUEjK0V3PH8+x3HSUkyCsdnMnFMBmt3VtPc2sZnZuZz2+Wns2jWxAHtCxeR409jCMNFawus+gn8190wvgiuvA+mnTOgX3G4uZV/fGo9v1+3B4D8nPSg3/6TBXz65HwmZMe+Hz5aZV0TH1fWs6Oynh3heEBZ1WGKC8dy3aeKjvtMShHpPY0hjCTu8NzfwbrfwFlfg/9+B6QP7Im1/FAj315WwobdNfzlwk9w6RmTOfWEMb2+OyM/J538nHTOKZowoOUTkaGngDAcrL4/CAYLb4WLlg744TfuruFbj5RwqLGFe6+bx2dnTxrw7xCRkS+uKYxmttjMtprZdjO7Ncb26Wa20szWm9lrZlYYte1nZrbJzLaY2f+x8FaSMN9WM1sbviYOXLVGkJ1vwQu3wimLYeEtA374lzbt40v//iZJBk/+5XkKBiJyVD22EMwsGfgV8FmgDFhtZs+5++aobHcBy9z9ETNbBNwBXGdm5wHnA3PDfP8FLAReCz9/1d1H6aBAHGr3wRNfC24NXfIf8c0UjlNjSyv/8fpH/OvKbcwtHMd9Xztbd8+IyDHF02U0H9ju7h8BmNljwOVAdECYDXwnfL8KeCZ870AGkEYwVz0VKO9/sUeBSHMQDJpq4bpnIHPcMbPXNUX40/ZKXt9Wwbs7q5lfND7mIK6784f1e7nzhfcpqzrM5cVTuPOquWSkDt/bPUVkeIgnIEwFdkV9LgMWdMmzDrgK+AWwBMg1szx3f9PMVgF7CQLCL919S9R+D5lZK/AU8GMfSbc89deL34Vdb8MXH4JJs2NmaWxpZdmbO3j1/f2U7Kgi0uZkpyVz2tSxLH9nF4+8WcqnT87na5+azp+dOol1ZdX8+A+beXdnNbNOyOXRby3g/KiJVyIixxJPQIh1G0rXE/dNwC/N7HrgDWA3EDGzk4FTgfYxhZfN7AJ3f4Ogu2i3meUSBITrgGXdvtzsBuAGgBNPPDGO4o4A65+A1ffBeX8Lp18ZM0trm/N3y9/jpc3lnDp5DN/6zEksPKWAs6ePJy0licq6Jh5fvYv/fKuUG369hvycNCrrminITednV83lqrMLNQ9ARHqlx3kIZvYp4Ifu/t/Dz0sB3P2Oo+TPAd5390IzuxnIcPd/Drf9AGh095912ed6YJ6733issoyKeQgNB+H/ng35M+H6FTEnnbk7P3h2E79+q5R/unQ2Xz9/xlEPF2lt45Ut5Tz97m5mTR7DX1xwkpZcEJFOBnIewmpgppnNILjy/wrwP7p8WT5w0N3bgKXAg+GmncC3zewOgpbGQuBfzSwFGOfulWaWCnwBeCW+qo1wq26Hxmq45OdHnYF8z+sf8uu3SvmLC046ZjAASElOYvHpk1l8+vFfGldERrceb2tx9whwI/AisAV4wt03mdltZnZZmO1CYKuZbQMmAbeH6U8CHwIbCMYZ1rn77wmeYfGima0H1hIEmvsGrFbD1d71UPIgnPNtOOH0mFmeWlPGz17YymVnTOGWxbMGuYAiksi0dMVgcYcHF8OB7fC3a2LeVfTGtgq+8fBq5s+YwENfP2dYLwQnIiOHlq4YbtY/Abvegsv+b8xgsOr9/dz4m3c5eWIO/37d2QoGIjLoFBAGQ+MhePn7MOUsKL6206b39x3i9v+3hT9+UMlJBdk88o35jMnQw1dEZPApIAyG1++Euv1wzfKO2cgVtU38/OVtPL56JznpKXz/C7O57tzpMR9DKCIyGBQQjreKrfD2v8NZ18HUswF4du1uvvv0Bpoibfz5eUX83aKZjI9j6WkRkeNJAeF42v4KPH8LpGXDn/0TAGt3VXPzb9czp3As//LFuZxUoOcHiMjwoIBwPOzfAi/97yAgjC+Cq5dBdj6VdU381X+uYeKYdO7/2jy1CkRkWFFAGEh1FfDaT2DNw5CWC5+7HeZ/G1LSibS28TePvsvB+mae+qvzFAxEZNhRQBgoh/bAvRdBQ2Uw8WzhLZCd17H5juff5+2PD3L3l8/g9Kljh7CgIiKxKSAMhEjTkaWsv/0qTD6j0+Zn1+7mgf/6mOvPK2LJmYVHOYiIyNBSQBgIL9wKZavhS490CwZb9h7ilqfWM79oAt+75NQhKqCISM9003t/vfvrYH2i8/8eTrui06aPK+v5xsOrGZuZyi+/eiapyfp1i8jwpRZCf+x+F/7f/4KTLoRFP+i06cOKOq659y0ibc6j31qgx1eKyLCngNBX9ZXw+HWQMxGuerDTUtYflNdyzX1vA85jN5zLKZNyh66cIiJxUkDoi7Y2ePLrUF8B33yx091EW/fV8tX738LMWP7tczl5ooKBiIwMCgh9sest+PgNuPhfYMqZHclb9h7iq/e/TWqy8Ztvn8snNAtZREYQBYS+WP8EpGZB8ZEHxzVH2vj6Q6tJT0li+bfPpSg/ewgLKCLSewoIvRVphs3PwKxLIP1IC+D5jXvZd6iRh79+joKBiIxIug+yt7a/AoerYM7VnZKXvVlKUV4WF8wsGKKCiYj0jwJCb61/HLLy4BMXdSRt2lPDmtIqrj13OklJNoSFExHpOwWE3mg8BNtegNOuhOQjTzX79ZulZKQm8aWzpw1h4URE+kcBoTe2/B4ijTD3SHdRTUMLz6zdzRXFUxmbpUdfisjIpYDQGxueCJ5vUHhOR9Jv1+yisaWN6z41fejKJSIyABQQ4lW7L5h7MOdLYME4QVub8+u3Sjl7+nhOm6IlrUVkZFNAiNfGp8DbOt1d9MYHFZQeaOBrah2IyCiggBCv9Y/D5GIoOKUj6ddvlpKfk87Fp08ewoKJiAwMBYR4VGyDves6DSbvOtjAq1v3c838aaSl6NcoIiOfzmTx2PAEWBKcflVH0n++XUqSGf9jwYlDWDARkYGjpSu6OlwFrZGoBIcNv4UZF0DuCUCwbtHjq3fx2VMnMXls5tCUU0RkgMUVEMxsMfALIBm4391/2mX7dOBBoAA4CFzr7mXhtp8BlxC0Rl4G/t7d3czOBh4GMoEV7ekDUak+2/o8LP9K7G0X/GPH2w27q6luaOHy4imDVDARkeOvx4BgZsnAr4DPAmXAajN7zt03R2W7C1jm7o+Y2SLgDuA6MzsPOB+YG+b7L2Ah8BpwD3AD8BZBQFgMPD8Qleqz/WGVLv5Z0EXULjWz0/jBmtIqAOYVTRjM0omIHFfxtBDmA9vd/SMAM3sMuByIDgizge+E71cBz4TvHcgA0gADUoFyM5sMjHH3N8NjLgOuYKgDQt1+SB8LC/7imNlKdlQxPS+Lgtz0QSqYiMjxF8+g8lRgV9TnsjAt2jqgfcR1CZBrZnnhCX8VsDd8vejuW8L9y3o4JgBmdoOZlZhZSUVFRRzF7Ye6csg59mql7s6a0irOnj7++JZFRGSQxRMQYi3f2bWv/yZgoZm9R9AltBuImNnJwKlAIcEJf5GZXRDnMYNE93vdfZ67zysoOM5LS9fth5xJx8yy40ADB+qbFRBEZNSJJyCUAdHLeBYCe6IzuPsed7/S3c8Evhem1RC0Ft5y9zp3ryPoEjo3PGbhsY45JOrKIWfiMbN0jB9M1/iBiIwu8QSE1cBMM5thZmnAV4DnojOYWb5ZxyjsUoI7jgB2ErQcUswslaD1sMXd9wK1ZnaumRnwNeDZAahP/8TRQlhTepAxGSnMnKjnJYvI6NJjQHD3CHAj8CKwBXjC3TeZ2W1mdlmY7UJgq5ltAyYBt4fpTwIfAhsIxhnWufvvw21/BdwPbA/zDO2AcnMDNB3qsYVQsqOKs6aP14NwRGTUiWsegruvILg1NDrtB1HvnyQ4+XfdrxWIecuOu5cAp/emsMdV/f7g5zFaCNUNzXywv07zD0RkVNLSFe3qeg4I7+4Mxg/O0oCyiIxCCgjt6sqDn8foMlpTWkVyklE8bdwgFUpEZPAoILTrCAhHbyGU7KjitCljyErTElAiMvooILSr2w8YZOXH3NzS2sa6smrNPxCRUUsBoV1dOWTnQ3Lsq/9New7R2NKmgCAio5YCQrse5iBoQpqIjHYKCO16mKW8pvQgU8dlcsLYjEEslIjI4FFAaHeMFoK7U7KjinlF6i4SkdFLAQHA/ZgthLKqw+yvbdL4gYiMagoIAI3V0Np81BZC+/iBAoKIjGYKCNDjLOWS0oPkpKcw64Qxg1goEZHBpYAAPc5SLtlRxZknjiNZC9qJyCimgADHbCEcamxha3ktZ52o7iIRGd0UECAqIHRvIazbVY27xg9EZPRTQICgyyg5DTK6L1q3vqwGgDMKtaCdiIxuCghwZA6CdR8j2FBWw/S8LMZmpQ5BwUREBo8CAhxzDsKG3TXMmTp2kAskIjL4FBDgqLOUK+ua2F19mLmFCggiMvopIMBRWwgbdgfjB3OmavxAREY/BYS2VmiojNlC2BAOKJ8+VRPSRGT0U0CorwRvi9lCWF9Ww0kF2eRmaEBZREY/BYRjPDpzw+5q3W4qIglDAeEos5TLDzVSfqhJdxiJSMJQQDjKOkbt4we6w0hEEoUCQntAyO4cENbvriHJYPYUDSiLSGJQQKjbD2m5kJbVKXlDWTUzJ+aSlZYyRAUTERlcCggx5iC4ezBDWd1FIpJA4goIZrbYzLaa2XYzuzXG9ulmttLM1pvZa2ZWGKZfZGZro16NZnZFuO1hM/s4alvxwFYtTjFmKe+taaSyrlnjByKSUHoMCGaWDPwKuBiYDVxjZrO7ZLsLWObuc4HbgDsA3H2Vuxe7ezGwCGgAXora7+b27e6+tv/V6YMYLYT2FU51h5GIJJJ4Wgjzge3u/pG7NwOPAZd3yTMbWBm+XxVjO8AXgefdvaGvhT0uYrQQNuyuJiXJOHWyBpRFJHHEExCmAruiPpeFadHWAVeF75cAuWaW1yXPV4DlXdJuD7uZ7jaz9FhfbmY3mFmJmZVUVFTEUdxeaDkMTTUxWwinTMolIzV5YL9PRGQYiycgxHqQsHf5fBOw0MzeAxYCu4FIxwHMJgNzgBej9lkKzALOASYAt8T6cne/193nufu8goKCOIrbCzEmpbUPKGv8QEQSTTz3VJYB06I+FwJ7ojO4+x7gSgAzywGucveaqCxXA79z95aoffaGb5vM7CGCoDK4YgSEXQcPU93QojuMRCThxNNCWA3MNLMZZpZG0PXzXHQGM8s3s/ZjLQUe7HKMa+jSXRS2GjAzA64ANva++P0UY5by+t3VAMzVktcikmB6DAjuHgFuJOju2QI84e6bzOw2M7sszHYhsNXMtgGTgNvb9zezIoIWxutdDv2omW0ANgD5wI/7VZO+iLGw3YayGtKSkzjlhJxBL46IyFCKaxquu68AVnRJ+0HU+yeBJ4+y7w66D0Lj7ot6U9Djom4/YJCd35G0vqyGWZNzSU/RgLKIJJbEnqlcVw5ZeZAcPO+grc3ZqGcoi0iCSvCA0HkOwo4D9dQ2RXSHkYgkpAQPCJ1nKW/ccwjQM5RFJDEldkCo79xC2FdzGIBpEzKHqkQiIkMmcQOCe9hldKSFcLC+hdRkIyddS16LSOJJ3IDQdAgijZ1aCAfrm5iQnUYwNUJEJLEkbkCIMUv5YH0L47PShqhAIiJDK4EDQvdZygfrm8jLUUAQkcSkgBDVQqhqUAtBRBJXAgeE9i6jIy2EA3VN5GUrIIhIYkrggFAOSamQOR6AltY2DjVGGK+AICIJKnEDQm150F0U3lFU1dAMoBaCiCSsxA0IdfsgN2r8oD54VINaCCKSqBI3INSWQ84JHR8P1gcthAkKCCKSoBI3IHRpISggiEiiS8yAEGmGhgOdWwgNCggiktgSMyDUh7ecRrcQ6oKAoHkIIpKoEjMg1LZPSjvSQqhqaGZMRgqpyYn5KxERScyzX92+4GdUC+FAfbO6i0QkoSVmQKgNA0J0C0EBQUQSXGIGhLpywCC7oCNJLQQRSXSJGRBq9wXBIPnIg3DUQhCRRJeYAaGuvNMqp+7OwfpmzVIWkYSWmAGhtvOktPrmVppb27SOkYgktMQMCHVdlq3QHAQRkQQMCG2twbMQoielta90qqeliUgCS7yA0HAAvLXLwnZNgFoIIpLY4goIZrbYzLaa2XYzuzXG9ulmttLM1pvZa2ZWGKZfZGZro16NZnZFuG2Gmb1tZh+Y2eNmNjhn49ruk9IOhktf52WnD0oRRESGox4DgpklA78CLgZmA9eY2ewu2e4Clrn7XOA24A4Ad1/l7sXuXgwsAhqAl8J97gTudveZQBXwzQGoT8/qui9b0dFCyE4dlCKIiAxH8bQQ5gPb3f0jd28GHgMu75JnNrAyfL8qxnaALwLPu3uDmRlBgHgy3PYIcEVvC98nR2khpCYbOekpR9lJRGT0iycgTAV2RX0uC9OirQOuCt8vAXLNLK9Lnq8Ay8P3eUC1u0eOcUwAzOwGMysxs5KKioo4ituDuu7LVhysb2JCdhoWPk5TRCQRxRMQYp0lvcvnm4CFZvYesBDYDbSf7DGzycAc4MVeHDNIdL/X3ee5+7yCgoJYWXqnthwyxkJqRkfSwfoWDSiLSMKLp4+kDJgW9bkQ2BOdwd33AFcCmFkOcJW710RluRr4nbu3hJ8rgXFmlhK2Erod87ip29epdQDB0te65VREEl08LYTVwMzwrqA0gq6f56IzmFm+mbUfaynwYJdjXMOR7iLc3QnGGr4YJv058Gzvi98HteWdxg8geHymWggikuh6DAjhFfyNBN09W4An3H2Tmd1mZpeF2S4EtprZNmAScHv7/mZWRNDCeL3LoW8B/sHMthOMKTzQr5rEK0YL4WB9s5atEJGEF9dtNe6+AljRJe0HUe+f5MgdQ1333UGMAWN3/4jgDqbB496thdDS2kbN4RYtbCciCS+xZio3VkNrU6cWQnVD+6Q0BQQRSWyJFRDan6WcG33LabiwnQKCiCS4xAoIHXMQoielBQFBD8cRkUSXWAHhGC0EBQQRSXSJFRBitRAaFBBERCDRAkJtOaRmQXpuR5IejiMiEkisgFC3L2gdRK1ZVNXQzJiMFFKTE+tXISLSVWKdBWvLO3UXARyob1Z3kYgIiRYQ6vZ1W7aiSgFBRARItIBQW95t2Qq1EEREAokTEJrroblWLQQRkaNInIBQ2/3BOO4erHSqgCAikkABof1ZylEthPrmVppb27SOkYgIiRgQoloIVfWagyAi0i5xAkKMZSsOhAFBT0sTEUmkgFC3D5JSIHNCR5JaCCIiRyROQGiflJZ0pModLYTs9KEqlYjIsJE4AaF92YooHS2E7NShKJGIyLCSOAGhtrzT+AEELYS05CRy0uN6kqiIyKiWOAHhKC2E8dmpWNRidyIiiSoxAkKkGRoOxGwhTND4gYgIkCgBoX5/8LNrC6GhmQkaPxARARIlIMSYgwDB4zPVQhARCSRGQIjx6EwIA0KWWggiIpAoAaF9YbuoFkJLaxs1h1u0sJ2ISCgxAkJdOWCQPbEjqbqhBUAL24mIhBIjINTug+x8SD4y3+Bgx6Q0BQQREYgzIJjZYjPbambbzezWGNunm9lKM1tvZq+ZWWHUthPN7CUz22Jmm82sKEx/2Mw+NrO14at4oCrVTV33J6W1BwQ9HEdEJNDjFF0zSwZ+BXwWKANWm9lz7r45KttdwDJ3f8TMFgF3ANeF25YBt7v7y2aWA7RF7Xezuz85EBU5pnO+Bc11nZIUEEREOotnzYb5wHZ3/wjAzB4DLgeiA8Js4Dvh+1XAM2He2UCKu78M4O6dz8qDZeZnuyUdbFBAEBkJWlpaKCsro7GxcaiLMuxlZGRQWFhIamrf7p6MJyBMBXZFfS4DFnTJsw64CvgFsATINbM84BSg2syeBmYArwC3untruN/tZvYDYGWY3tSnWvTBwTotfS0yEpSVlZGbm0tRUZGWmTkGd+fAgQOUlZXnBTcxAAAM4ElEQVQxY8aMPh0jnjGEWP8C3uXzTcBCM3sPWAjsBiIEAecz4fZzgJOA68N9lgKzwvQJwC0xv9zsBjMrMbOSioqKOIobn0ONLeSkp5CanBjj6iIjVWNjI3l5eQoGPTAz8vLy+tWSiudsWAZMi/pcCOyJzuDue9z9Snc/E/hemFYT7vueu3/k7hGCrqSzwu17PdAEPETQNdWNu9/r7vPcfV5BQUEvq3d0TZFWMlIVDERGAgWD+PT39xTPGXE1MNPMZphZGvAV4Lkuhcg3s/ZjLQUejNp3vJm1n8kXEY49mNnk8KcBVwAb+1OR3mpqaSNNrQMRkQ49nhHDK/sbgReBLcAT7r7JzG4zs8vCbBcCW81sGzAJuD3ct5Wgu2ilmW0g6H66L9zn0TBtA5AP/HjAahWHpkgb6anJg/mVIjKKfOtb32Lz5s09Z+yHz3/+81RXV3dL/+EPf8hdd9014N8X15Nh3H0FsKJL2g+i3j8JxLx9NLzDaG6M9EW9KukAa4q0kp6iFoKI9M39999/3L9jxYoVPWcaQAn7qLCmSJsCgsgI86Pfb2LznkMDeszZU8bwT5eedsw89fX1XH311ZSVldHa2sr3v/997rnnHu666y7mzZvHAw88wJ133smUKVOYOXMm6enp/PKXv+T6668nMzOT999/n9LSUh566CEeeeQR3nzzTRYsWMDDDz8MwPLly/nJT36Cu3PJJZdw5513AlBUVERJSQn5+fncfvvtLFu2jGnTplFQUMDZZ589oL8HSJSlK2JoamkjPUVdRiLSsxdeeIEpU6awbt06Nm7cyOLFizu27dmzh3/+53/mrbfe4uWXX+b999/vtG9VVRWvvvoqd999N5deeinf+c532LRpExs2bGDt2rXs2bOHW265hVdffZW1a9eyevVqnnnmmU7HWLNmDY899hjvvfceTz/9NKtXrz4u9UzYFkJzaxtZaQoIIiNJT1fyx8ucOXO46aabuOWWW/jCF77AZz7zmY5t77zzDgsXLmTChAkAfOlLX2Lbtm0d2y+99FLMjDlz5jBp0iTmzJkDwGmnncaOHTsoLS3lwgsvpP0uyq9+9au88cYbXHHFFR3H+OMf/8iSJUvIysoC4LLLLuN4SNiA0BRpZVymnoUgIj075ZRTWLNmDStWrGDp0qV87nOf69jm3nVaVmfp6cFDuJKSkjret3+ORCKkpMR3Gh6MW28Tu8tI8xBEJA579uwhKyuLa6+9lptuuol33323Y9v8+fN5/fXXqaqqIhKJ8NRTT/Xq2AsWLOD111+nsrKS1tZWli9fzsKFCzvlueCCC/jd737H4cOHqa2t5fe///2A1KurBG4haAxBROKzYcMGbr75ZpKSkkhNTeWee+7hpptuAmDq1Kl897vfZcGCBUyZMoXZs2czduzYuI89efJk7rjjDi666CLcnc9//vNcfvnlnfKcddZZfPnLX6a4uJjp06d36rIaSNZTc2c4mTdvnpeUlAzIsRb85BUu+uREfnpVtztiRWQY2bJlC6eeeupQF+OY6urqyMnJIRKJsGTJEr7xjW+wZMmSISlLrN+Xma1x93k97ZuwfSa67VREBsoPf/hDiouLOf3005kxY0anAeGRJGG7jJojbaQpIIjIADges4aHQsKeETWGICLSWUIGhEhrG61tri4jEZEoCXlGbIoET/HUbaciIkck5BmxIyCoy0hEpEOCBoTgCZ7qMhKRnlRXV/Nv//Zvfd7/wgsvZKBulz/eEvKM2By2EHSXkYj0pL8BYSRJyNtO1WUkMkI9fyvs2zCwxzxhDlz806NuvvXWW/nwww8pLi7moosuYv369VRVVdHS0sKPf/xjLr/8cnbs2MHFF1/Mpz/9af70pz8xdepUnn32WTIzMwH47W9/y1//9V9TXV3NAw88cNxmGvdXYgaElvaAoBaCiBzbT3/6UzZu3MjatWuJRCI0NDQwZswYKisrOffccztWHv3ggw9Yvnw59913H1dffTVPPfUU1157LQCRSIR33nmHFStW8KMf/YhXXnllKKt0VIkZENrHEHSXkcjIcowr+cHg7nz3u9/ljTfeICkpid27d1NeXg7AjBkzKC4uBuDss89mx44dHftdeeWVMdOHmwQNCOoyEpHee/TRR6moqGDNmjWkpqZSVFREY2MjQKelrZOTkzl8+HDH5/ZtycnJRCKRwS10LyTkJXJ7C0GDyiLSk9zcXGprawGoqalh4sSJpKamsmrVKkpLS4e4dAMrIVsIzRGNIYhIfPLy8jj//PM5/fTTOeecc3j//feZN28excXFzJo1a6iLN6ASMiA0KSCISC/85je/6THPxo0bO963PysB4LXXXut4n5+fP6zHEBLyjNhxl1GqxhBERNolZkDQTGURkW4S8oyoLiORkWUkPdlxKPX395SQZ8QmLV0hMmJkZGRw4MABBYUeuDsHDhwgIyOjz8dI6EHltGQFBJHhrrCwkLKyMioqKoa6KMNeRkYGhYWFfd4/QQNCK+kpSZjZUBdFRHqQmprKjBkzhroYCSGuS2QzW2xmW81su5ndGmP7dDNbaWbrzew1MyuM2naimb1kZlvMbLOZFYXpM8zsbTP7wMweN7O0gapUT5pa2jR+ICLSRY9nRTNLBn4FXAzMBq4xs9ldst0FLHP3ucBtwB1R25YB/+LupwLzgf1h+p3A3e4+E6gCvtmfivRGU6RNt5yKiHQRz2XyfGC7u3/k7s3AY8DlXfLMBlaG71e1bw8DR4q7vwzg7nXu3mBBX80i4Mlwn0eAK/pVk15oirRq/EBEpIt4xhCmAruiPpcBC7rkWQdcBfwCWALkmlkecApQbWZPAzOAV4BbgfFAtbtHoo45NdaXm9kNwA3hxzoz2xpHmWPJByo7HXtpH480PHSrzwg2muoCo6s+o6kukLj1mR7PweIJCLFGXrve/3UT8Eszux54A9gNRMLjfwY4E9gJPA5cDzwXxzGDRPd7gXvjKOcxmVmJu8/r73GGi9FUn9FUFxhd9RlNdQHVpyfx9JuUAdOiPhcCe6IzuPsed7/S3c8Evhem1YT7vhd2N0WAZ4CzCCLaODNLOdoxRURkcMUTEFYDM8O7gtKAr9DlCt/M8s2s/VhLgQej9h1vZgXh50XAZg9mmKwCvhim/znwbN+rISIi/dVjQAiv7G8EXgS2AE+4+yYzu83MLguzXQhsNbNtwCTg9nDfVoLupJVmtoGg++m+cJ9bgH8ws+1AHvDAgNUqtn53Ow0zo6k+o6kuMLrqM5rqAqrPMZmmg4uICCToWkYiItKdAoKIiAAJEhB6WnpjODOzB81sv5ltjEqbYGYvh8t+vGxm44eyjL1hZtPMbFW4lMkmM/v7MH3E1cnMMszsHTNbF9blR2H6kC3LMhDMLNnM3jOzP4SfR2R9zGyHmW0ws7VmVhKmjbi/s3ZmNs7MnjSz98P/P58a6PqM+oAQ59Ibw9nDwOIuabcCK8NlP1aGn0eKCPC/wqVMzgX+Jvz3GIl1agIWufsZQDGw2MzOZQiXZRkgf09wA0m7kVyfi9y9OOpe/ZH4d9buF8AL7j4LOIPg32hg6+Puo/oFfAp4MerzUmDpUJerl3UoAjZGfd4KTA7fTwa2DnUZ+1G3Z4HPjvQ6AVnAuwSz+CsJlmzp9vc33F8Ec4JWEtwi/geCOwNHZH2AHUB+l7QR+XcGjAE+JrwR6HjVZ9S3EIi99EbMZTJGkEnuvhcg/DlxiMvTJ+HKt2cCbzNC6xR2r6wlWLTxZeBD4lyWZZj6V+Afgbbwcx4jtz4OvGRma8IlcGCE/p0BJwEVwENhd979ZpbNANcnEQJCPEtvyCAzsxzgKeB/uvuhoS5PX7l7q7sXE1xZzwdOjZVtcEvVN2b2BWC/u6+JTo6RdUTUBzjf3c8i6C7+GzO7YKgL1A8pBKs83OPBihD1HIfurkQICD0uvTEClZvZZIDw5/4e8g8rZpZKEAwedfenw+QRXSd3rwZeIxgXGanLspwPXGZmOwhWNV5E0GIYkfVx9z3hz/3A7wgC9kj9OysDytz97fDzkwQBYkDrkwgBocelN0ag5wiW+4ARtuxHuPT5A8AWd/951KYRVyczKzCzceH7TOC/EQz0jchlWdx9qbsXunsRwf+TV939q4zA+phZtpnltr8HPgdsZAT+nQG4+z5gl5l9Mkz6M2AzA12foR4sGaQBmc8D2wj6d7831OXpZdmXA3uBFoKrhG8S9OuuBD4If04Y6nL2oj6fJuhyWA+sDV+fH4l1AuYC74V12Qj8IEw/CXgH2A78Fkgf6rL2oW4XAn8YqfUJy7wufG1q/38/Ev/OoupUDJSEf2/PEDxGYEDro6UrREQESIwuIxERiYMCgoiIAAoIIiISUkAQERFAAUFEREIKCCIiAiggiIhI6P8D565DhNy8+j4AAAAASUVORK5CYII=\n",
      "text/plain": [
       "<Figure size 432x288 with 1 Axes>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "import matplotlib.pyplot as plt\n",
    "\n",
    "accuracies_sigmoid = [\n",
    "    0.9416, 0.9649, 0.9738, 0.9773, 0.9804, 0.9821, 0.9828, 0.9837, 0.9841, 0.9849, 0.9851, 0.9848, 0.9852, 0.9858,\n",
    "    0.986, 0.9862, 0.986, 0.9859, 0.9862, 0.9863, 0.9864, 0.9866, 0.9867, 0.9867, 0.9867, 0.9868, 0.9865, 0.9864,\n",
    "    0.9865, 0.9866, 0.9866, 0.9866, 0.9867, 0.9867, 0.9868, 0.9868, 0.9868, 0.9867, 0.9867, 0.9867, 0.9867, 0.9867,\n",
    "    0.9867, 0.9868, 0.9869, 0.9869, 0.9868, 0.9868, 0.9867, 0.9868, 0.9869, 0.987, 0.987, 0.987, 0.987, 0.987, 0.9871,\n",
    "    0.9871, 0.9872, 0.9872\n",
    "]\n",
    "\n",
    "accuracies_tanh = [\n",
    "    0.9654, 0.9727, 0.9803, 0.9803, 0.9823, 0.9832, 0.9836, 0.9848, 0.9857, 0.9862, 0.9862, 0.9862,\n",
    "    0.9863, 0.9864, 0.9865, 0.9864, 0.9865, 0.9866, 0.9866, 0.9866, 0.9866, 0.9866, 0.9867, 0.9867, 0.9868, 0.9868,\n",
    "    0.9867, 0.9867, 0.9867, 0.9867, 0.9868, 0.9868, 0.9868, 0.9867, 0.9867, 0.9867, 0.9867, 0.9867, 0.9867, 0.9867,\n",
    "    0.9867, 0.9867, 0.9867, 0.9867, 0.9868, 0.9868, 0.9868, 0.9868, 0.9868, 0.9868, 0.9868, 0.9868, 0.9868, 0.9868,\n",
    "    0.9868, 0.9868, 0.9868, 0.9868, 0.9868, 0.9868\n",
    "]\n",
    "\n",
    "plt.ylim(bottom=0.96, top=0.99)\n",
    "plt.plot(accuracies_sigmoid, label=\"sigmoid\")\n",
    "plt.plot(accuracies_tanh, label=\"tanh\")\n",
    "plt.legend()\n",
    "plt.show()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Indeed, it appears that $\\tanh$ learns a little faster in the beginning, but final accuracies are very close.\n",
    "\n",
    "Why does it learn faster?\n",
    "\n",
    "A first possible reason may be that since $\\tanh'(z) = 4 \\sigma'(2z)$, when $z$ is small, $\\tanh'$ is closer to $1$ than $\\sigma'$ because of the factor $4$. But when $z$ is big, because of the exponential, $4\\sigma'(2z)$ becomes much smaller than $\\sigma'(z)$.\n",
    "\n",
    "And recall from Chapter 5 that small values of derivatives were a cause of the vanishing gradient problem. So the $\\tanh$ function seems better for small values of $z$, and worse for large ones.\n",
    "\n",
    "The benefit with small values of $z$ (which can be considered to be quite frequent, especially if there is some regularization) may outweigh the cost on large values.\n",
    "\n",
    "Another reason might be that $\\tanh$ is symmetric about the origin, while $\\sigma$ is always positive. In [Efficient BackProp (1998)](http://yann.lecun.com/exdb/publis/pdf/lecun-98b.pdf), LeCun et al. write:\n",
    "\n",
    "> Convergence is usually faster if the average of each input variable over the training set is close to zero. To see this, consider the extreme case where all the inputs are positive. Weights to a particular node in the first weight layer are updated by an amount proportional to $\\delta x$ where $\\delta$ is the (scalar) error at that node and $x$ is the input vector [...]. When all of the components of an input vector are positive, all of the updates of weights that feed into a node will be the same sign (i.e. sign($\\delta$)). As a result, these weights can only all decrease or all increase *together* for a given input pattern. Thus, if a weight vector must change direction it can only do so by zigzagging which is inefficient and thus very slow. [...] Sigmoids that are symmetric about the origin are preferred for the same reason that inputs should be normalized, namely, because they are more likely to produce outputs (which are *inputs* to the next layer) that are on average close to zero.\n",
    "\n",
    "Let's be more visual:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 3,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "image/png": "iVBORw0KGgoAAAANSUhEUgAAAYYAAAD8CAYAAABzTgP2AAAABHNCSVQICAgIfAhkiAAAAAlwSFlzAAALEgAACxIB0t1+/AAAADl0RVh0U29mdHdhcmUAbWF0cGxvdGxpYiB2ZXJzaW9uIDIuMi4yLCBodHRwOi8vbWF0cGxvdGxpYi5vcmcvhp/UCwAAIABJREFUeJzs3Xd4lFW+wPHvSSekkEZMISQgnVBCICjSVBCRqoK66hUVsWzVlUW9rt1d3HXdu2vBgr2ABRVUbEhTEUNCDTUEAiSB9ErqzJz7xzsTkjBJJpnJtJzP8+R5J289gUl+c9rvCCkliqIoimLi4egCKIqiKM5FBQZFURSlGRUYFEVRlGZUYFAURVGaUYFBURRFaUYFBkVRFKUZFRgURVGUZlRgUBRFUZpRgUFRFEVpxsvRBeiM8PBwGR8f7+hiKG4qPT29SEoZYe/nqve10tUsfW+7ZGCIj48nLS3N0cVQ3JQQ4oQjnqve10pXs/S9rZqSFEVRlGZUYFAURVGaUYFBURRFacYl+xjMaWhoICcnh9raWkcXxan5+fkRGxuLt7e3o4uiKIqTcpvAkJOTQ2BgIPHx8QghHF0cpySlpLi4mJycHBISEhxdHEVRnJRNmpKEEG8IIQqEEBmtHBdCiP8KIY4KIfYKIZKaHLtFCJFp/Lqls2Wora0lLCxMBYU2CCEICwtTtSobsOY9ryjOzlZ9DG8BM9o4fiUwwPi1BFgBIIQIBR4FUoBxwKNCiJDOFkIFhfapfyObeYtOvOcVxRXYpClJSrlVCBHfxilzgXekto7odiFELyFEFDAF+F5KWQIghPge7ZdtlS3KpbgQKaGhGmoroLYc6ipAVwf6OtDVN9/q67XzpQRpaP5Fy33y3BbjNQD+YTD+LiuK27n3vJTydKcfqpwjJWSsgeBYiBvv6NK4HXv1McQAp5p8n2Pc19r+8wghlqB98iIuLq5rSmljixcv5r777mPo0KFd9oyZM2fywQcf0KtXr2b7H3vsMQICArj//vu77NkdVnkGCg5AwUEozoKKPKjIgYrTUFsGBp1dimFAcMa7D9FWBAYLtPbebhYYXPF97RS2PAOb/669vvkz6H+pY8vjZuwVGMy1X8g29p+/U8pXgVcBkpOTzZ7jbFauXNnlz1i/fn2XP6PTaivgyLdwfDMc3wplJ88d6xECQbEQFA0xY6BHKPgFgV+w9uUbBF6+4OkLXj7GrS8ldZBVUs/xompOldaRW15HXnkdOWW1lNboMCAw4IFENL728/YkuIcPgT28CfTzJtDPi/iwnjzWtT+9Re9tV3xfO1xNGWx7QQsGRZmw+RkVGGzMXoEhB+jT5PtYIM+4f0qL/ZvtVCabOnv2LAsXLiQnJwe9Xs9f//pXVqxYwbPPPktycjKvv/46zzzzDNHR0QwYMABfX19eeOEFFi1aRI8ePTh06BAnTpzgzTff5O233+aXX34hJSWFt956C4BVq1bxt7/9DSklV111Fc888wxwLo1CeHg4Tz/9NO+88w59+vQhIiKCMWPGOOYf41QqbF8Bh9eDrhb8ekH8JZByN1wwHHoPhZ7h7d6mqKqOXSfL2HmylN0nyzicX0nJ2frG437eHsT06kFsSCiT4noQ06sHvQN9CQvwIaynL6E9fQgL8MHfxyGD71p7zyvWOvQV1FfC1Ifh5Db47mEoOQah/RxdMrdhr9+YdcDvhBCr0Tqay6WUp4UQ3wJ/a9LhPB140NqHPf7Ffg7kVVh7m2aGRgfx6OxhrR7/5ptviI6O5quvvgKgvLycFSu0/sa8vDyefPJJdu7cSWBgIJdeeikjR45svLa0tJSNGzeybt06Zs+ezc8//8zKlSsZO3Ysu3fvpnfv3ixbtoz09HRCQkKYPn06n3/+OfPmzWu8R3p6OqtXr2bXrl3odDqSkpLsHxhO79F+SY9v1YLB6JshcQHEJoOHZ7uXV9fr2Ha0mM1HCvgps4js4moAvDwEQ6ODmD40kgGRgQyMDGBgZCC9A32duTPd7HvewWVyD5nfQcAFEJMEPXpp77nMDZCyxNElcxs2CQxCiFVon/zDhRA5aCONvAGklC8D64GZwFGgGrjVeKxECPEksMN4qydMHdGuJjExkfvvv59ly5Yxa9YsJk6c2HgsNTWVyZMnExoaCsCCBQs4cuRI4/HZs2cjhCAxMZHIyEgSExMBGDZsGNnZ2Zw4cYIpU6YQEaElRbzxxhvZunVrs8Dw448/Mn/+fPz9/QGYM2dOl//MjRpqtPbebS+AfyhMfwrG3Aq+Ae1eWlOv57sDZ/h8Vy4/Hy2mXm/A38eTi/uHccO4OJL6hpAYE4yfd/uBxZ46+55XrGQwwLFNMGQ2CAFh/bWaQtYPKjDYkK1GJd3QznEJ/LaVY28Ab9iiHCZtfbLvKgMHDiQ9PZ3169fz4IMPMn369MZjUrbddOzr6wuAh4dH42vT9zqdDi8vy/6bHPLpuewUfHijVltI+h+Y9oTWf9CO3afKePeXE3yTcZqz9Xqigv24+aK+TB3Um7EJIfh6OVcgaMma97xiheKj2qi1uIvO7Yu7SOvLklILForVVK4kG8nLy8Pf35+bbrqJ+++/n507dzYeGzduHFu2bKG0tBSdTseaNWs6dO+UlBS2bNlCUVERer2eVatWMXny5GbnTJo0ic8++4yamhoqKyv54osvbPJztanwMKy8HEqOww0fwpzn2wwKBoNk/b7TXP3Sz8x78We+3X+GWSOiWXXHeH5edil/nTWUSwaEO31QUBwoN13bxjRpJo1Jguqi5oMbFKu4TUoMR9u3bx9Lly7Fw8MDb29vVqxY0ThUNCYmhoceeoiUlBSio6MZOnQowcHBFt87KiqKv//970ydOhUpJTNnzmTu3LnNzklKSuK6665j1KhR9O3bt1lTVpcoOgpvXQUIuO1biGx9SK6Ukh8OFvDsd4c5dKaSuFB/Hp09lAXJfQjwVW9BpQPydoJ3TwgfeG6fKUjk7YSQvo4pl5sR7TVzOKPk5GTZckGTgwcPMmTIEAeVqH1VVVUEBASg0+mYP38+t912G/Pnz3dIWaz+t6ougZWXaVX6276F8AGtP+t0BY+szWBHdinxYf7cO20gs0ZE4+nhvFV+IUS6lDLZ3s81975WWnhzpjbf5fbvzu3T1cHTUTDxPrj0YceVzQVY+t5WH9fs5LHHHmPDhg3U1tYyffr0Zh3HLsVggDWLoTwXbvmi1aBwtk7H/204whs/ZxPcw5u/zU9kQXIs3p6q9VLpJCm1yZFDZjff7+WrdUIXHHRMudyQCgx28uyzzzq6CLaR+oo2AuSq5yAuxewpu0+V8cfVuzhRXM0N4/rwlysGE9LTx84FVdzO2UKoKYHeZmq7EYO1WfWKTajAoFiu5Bh8/ygMvBKSbzvvsMEgeXlrFs99d4TIID8+XDKelH5hDiio4pZMNYKIQecfixgMh76Ehlrw9rNvudyQCgyK5b5+ADy9Yda/zxsWWFWn40+rd7HhYAFXjYjib/MTCe6hFgNSbKj4qLZt2vFsEjFIS5ZYmg29B9u1WO5IBQbFMke+g8xvtclrQVHNDp0qqeaOd9LILKjiibnDuHl8X2eekay4qtJs8PSBwOjzj4XEnztHBQarqcCgtE9K2PSU9suX0jwjaUZuObe8kUq93sBbt45l4oAIx5RRcX+l2dCrL3iYGcDQNDAoVlNDRGykrKyMl156qdPXT5kyBacdqnj4a21m86S/aE1JRjtPlnLDa9vx8/bks3smqKCgdK2yE+cCQEv+YeAToJ2jWE0FBhuxNjA4LSlh6z8hJAFGXNe4+9djxdy88ldCe/rw4Z3jubB3+3mRFMUqpdmtT2ATQqtNqBqDTajAYCMPPPAAWVlZjBo1invvvZfLLruMpKQkEhMTWbt2LQDZ2dkMGTKEO+64g2HDhjF9+nRqamoa7/Hxxx8zbtw4Bg4cyI8//uioH6W5nDRtRulFvwVPreVx18lSFr25g6hePfjozouIDfF3cCEVt1dTqk2obK3GANoxFRhswj37GL5+AM7ss+09L0iEK5e3enj58uVkZGSwe/dudDod1dXVBAUFUVRUxPjx4xuznWZmZrJq1Spee+01Fi5cyJo1a7jpppsA0Ol0pKamsn79eh5//HE2bNhg25+hM1Jf0RbNGanljDtaUMmtb+2gd5AvH9yRQu9ANTRQsQPTH/z2AsOxTSqZng24Z2BwMCklDz30EFu3bsXDw4Pc3Fzy8/MBSEhIYNSoUQCMGTOG7Ozsxuuuvvpqs/sdpqoA9n8OY28H3wDyymq4+fVUvDw8ePc2FRQUO7I0MDRUaxPhAnrboVDuyz0DQxuf7O3h/fffp7CwkPT0dLy9vYmPj6e2thagWVptT0/PZk1JpmOenp7odPZZ/7hN+z4GQwOMuZWzdTpue2sHVbU6Vt85nrgw1Xyk2JEpMPRqI0le05FJKjBYxSZ9DEKIGUKIw0KIo0KIB8wc/7cQYrfx64gQoqzJMX2TY+tsUR5HCAwMpLKyEtBWb+vduzfe3t5s2rSJEydcdKTE3g8hahSG8EHc99FujuRX8tJNSQyLtjwzrKLYRGm2NvLIL6j1c9SQVZuxusYghPAEXgSmoa1zu0MIsU5K2Zi4REp5b5Pzfw+MbnKLGinlKGvL4WhhYWFMmDCB4cOHM3bsWA4dOkRycjKjRo1i8GAXnHBTcEgbojpjOc9vPMq3+/N5+Kohakiq4hilJ9quLQD0ijOem93lxXF3tmhKGgcclVIeAzCucTsXaC2j1Q1oyyC6nQ8++KDdczIyMhpfm9ZrANi8eXPj6/DwcMf3Mez7CIQnW3wm8e8NR7g6KYbbL0lwbJmU7qsiV8uH1BZvP/AP185VrGKLpqQY4FST73OM+84jhOgLJAAbm+z2E0KkCSG2CyFcNBe1Gzr4BbWxF/GHL3IZHhPE3+YnqjQXiuNUnIYgM6kwWgqK1s5VrGKLGoO5vxatrf5zPfCJlFLfZF+clDJPCNEP2CiE2CelzDrvIUIsAZYAxMXFWVtmpS1FR6HoCO8HXoZOb+CFG5Lw81bLbSoOUlsB9ZUWBoYYKM/p+jK5OVvUGHKAPk2+jwXyWjn3emBV0x1Syjzj9hiwmeb9D03Pe1VKmSylTI6IMN/O7Yqr0dmbRf9Gh78C4PXCITw9P5H48J5dXCpFaUOF8c9JkNmGiOaColVTkg3YIjDsAAYIIRKEED5of/zPG10khBgEhAC/NNkXIoTwNb4OBybQet9Em/z8/CguLlbBoQ1SSoqLi/Hza3v+QeXutewzxHPxmFHMG23BL6OidCXTH3qLagxR2mI+DTXtn6u0yuqmJCmlTgjxO+BbwBN4Q0q5XwjxBJAmpTQFiRuA1bL5X+4hwCtCCANakFredDRTR8TGxpKTk0NhYWHnf5huwM/Pj9jY2FaP15QX0bNwJ6m+1/H4nGF2LJmitKLS2GdgaVOS6ZrQfl1XJjdnkwluUsr1wPoW+x5p8f1jZq7bBiTaogze3t4kJKhRM9b6cu0qFiAZd9m19PR1z/mPiosxNSUFRrV9HpwLHhV5KjBYQSXRUxqlHi9Bl/kDNZ4BJI6d6ujiKIqmIlcbhurl2/65gU0Cg9JpKjAoANTU6/nLx7uZ7LUf7/6TGjOpKorDVeRZ1owE51YXVB3QVlGBQQHgX98dRpYeJ1oW4HXhpY4ujqKcU3HashFJAL6B4Bus5jJYSQUGhf155bzx83H+1N/4y9RviiOLoyjNVeSet854m9SQVaupwNDNGQyShz/PIMTfh6uCsqFnbwi70NHFUhRNQ402/NTSpiTQgojqY7CKCgzd3Idpp9h1soyHZg7BJ+9XiBuvFjlRnIdpqGpgBwJDYLQKDFZSgaEbK66qY/nXhxiXEMrVFwJlJ6HvxY4ulkuwINV8nBBikxBilxBirxBipiPK6fKqCrRtYKTl1wRGaov1GPTtn6uYpQJDN/bMN4c4W6fjqXnDESe3azvjxju2UC6gSar5K4GhwA1CiKEtTnsY+EhKORotG8BL9i2lm6jSVj4koAOBIeACkHqoLumaMnUDKjB0U+knSvkoLYfbJyYwMDIQTm4H754QaZP5hu6uMdW8lLIeMKWab0oCplVlgmk9f5jSFlONoUOBwbh6W9UZ25enm1CBoRsyGCRPfHmAyCBf/nDpAG3nqV8hNlnNX7CMJanmHwNuEkLkoGUF+L19iuZmqvJBeGirt1kq8IJz1yqdogJDN7R2Ty57TpXxlysGa2kvGmogf78WGBRLWJJq/gbgLSllLDATeFcIcd7vmxBiiXE9kjSV58uMqnzoGQEeHUj7bqoxVKrA0FkqMHQz1fU6nvn6MCNig5lvypx6Zp/WJhud5NjCuQ5LUs3fDnwEIKX8BfADwlveyJJ08t1aVcG5P/SWMjU7qRpDp6nA0M28suUYZypqeWTWUDw8jB98c9O1bcwYxxXMtViSav4kcBmAEGIIWmBQVYKOqsrvWP8CgE9P8AlUgcEKKjB0I3llNbyyNYtZI6JIjg89dyB3p5a5siOzS7sxKaUOMKWaP4g2+mi/EOIJIcQc42l/Bu4QQuxBW5xqkVSLhXRcVUHHAwNoQ1ZVYOg01dPYjfzjm0MYJDxwZYtF1fN2qmakDmov1bxxXZEJ9i6XWzEYOteUBFowUX0MnaZqDN3EzpOlfL47jyUT+xEb4n/uQE0ZFB+FGBUYFCdTWwaGhs7VGAJUjcEaNgkMFswCXSSEKBRC7DZ+LW5y7BYhRKbx6xZblEdpTkrJE18cICLQl7un9G9+8Mw+bRs9yv4FU5S2NM5h6GSNQQWGTrO6KanJLNBpaKM1dggh1plZovNDKeXvWlwbCjwKJKMN90s3XltqbbmUc9btyWP3qTL+ee2I81dly8/Qtmpim+JsOjPr2SQwEuqroK4KfANsW65uwBY1BktmgbbmCuB7KWWJMRh8D8ywQZkUo5p6Pcu/PsTwmCCuSTKz1vOZDG2ceEdy0SiKPXRm1rOJGrJqFVsEBktmgQJcY0wm9okQwjQG3NJrlU56desxTpfX8sisYeeGpzaVvw8ih9u/YIrSnsYaQ2eakkxpMQpsV55uxBaBwZJZoF8A8VLKEcAG4O0OXKudqGaIdtjp8hpe3pLFVYlRjEsIPf8EvQ4KDsEFKjAoTqgqH7z8wDeo/XNbCjClxVD5kjrDFoGh3VmgUspiKWWd8dvXgDGWXtvkHmqGaAf945vD6KU8f3iqSXEm6OtU/4LinExDVTuzPkhjU5KqMXSGLQJDu7NAhRBNZ07NQZsUBNoEoelCiBAhRAgw3bhPsdLOk6V8tiuXxZck0CfU3/xJZ4wdz6rGoDijzsx6NvEPA+EJlarG0BlWj0qSUuqEEKZZoJ7AG6ZZoECalHId8AfjjFAdUAIsMl5bIoR4Ei24ADwhpVRJ1K1kMJwbnnrP1DaW6czfB54+ED7QfoVTFEtVFUBoQueu9fDQahuqxtApNpn5bMEs0AeBB1u59g3gDVuUQ9Gs3ZPbODw1oOXw1KbOZEDEIPD0tl/hFMVSVfkQl9L56wMiVR9DJ6mZz26mafZUs8NTm8rPUP0LinPSN0B1ceebkkBNcrOCCgxu5uXNWZypqOXR2UPND081qSrUfmlU/4LijM4WAVKbY9NZgSpfUmepwOBGckqreWXrMeaMjGZMXzPDU5sq2K9tI4d1fcEUpaOsmfVsEhAJZwu1YdlKh6jA4EaWf30IIcxkTzWn8LC2jbDgXEWxN2tmPZsERAJSa5JSOkQFBjexI7uEL/ee5s5J/Ynu1aP9CwoPg1+wdb94itJVrJn1bNI4+1k1J3WUCgxuQKc38Ni6/UQF+3HX5P7tXwBQdATCB3Vu8pCidDWbBAY1ya2zVGBwA+9tP8H+vAoevmooPXwsXDS98BBEqPkLipOqKgDfYPC2oPbbGpVIr9NUYHBxBRW1/Ou7I0waGMHMxAssu6i6ROuUU/0LirOqyreutgCqKckKKjC4uKe+Okid3sATc4YhLG0WMnU8hw/quoIpijU6u9ZzUz49wSdQNSV1ggoMLuynzCLW7cnjnin9iQ/vafmFRaYRSaopSXFStqgxgDEthqoxdJQKDC6qTqfnkbUZxIf5W97hbFJ4BLx6QHBc1xROUaxlixoDGGc/qxpDR6nA4KJe2HiUY0VneXzucPy8LexwNik8BOEDtERjiuJs6s9CfaWqMTiQ+svggjJyy3lpcxbXJMUyeWAnUgYUHdGS5ymKM7LF5DYTVWPoFBUYXEyD3sDST/YS2tOHv84a0vEb1FVB+SkVGBTnZdPA0BvqyqGhxvp7dSMqMLiYFZuzOHi6gqfnDaeXv0/Hb1B0RNuqEUmKs7LF5DYTNcmtU1RgcCEZueU8vzGT2SOjmT7MwjkLLRVlalu1OI/irGyRQM9EBYZOsUlgEELMEEIcFkIcFUI8YOb4fUKIA0KIvUKIH4QQfZsc0wshdhu/1rW8VtFU1+v4w6pdhPX05cm5VmRELckCROdXxlKUrlZVAMIDeoZbfy81ya1TrF7BTQjhCbwITANygB1CiHVSygNNTtsFJEspq4UQdwP/AK4zHquRUo6ythzu7skvD3C8+CzvL07pXBOSSXEWBPcBL1/bFU5RbOlsAfiHg0cHR9uZo9JidIotagzjgKNSymNSynpgNTC36QlSyk1Symrjt9uBdpYWU5r6JuMMq1JPcdfk/lzc38pPUSXHIKyfbQqmKF3BVnMYwFjrEKopqYNsERhigFNNvs8x7mvN7cDXTb73E0KkCSG2CyHmtXaREGKJ8by0wsJC60rsQrKLzrL0kz2MiA3m3sut7BeQUmtKCu3ghDhFsSdbzXoGbT1z/zBVY+ggq5uSAHMJeqTZE4W4CUgGJjfZHSelzBNC9AM2CiH2SSmzzruhlK8CrwIkJyebvb+7qa7Xcee76Xh5CF78TRI+XlbG8eoSqC2HMBUYFCdWVWDbUXNqLkOH2aLGkAP0afJ9LJDX8iQhxOXA/wJzpJR1pv1Syjzj9hiwGRhtgzK5PCklf/lkL5kFlfz3htH0CfW3/qYlxniragxWa2/AhfGchcZBF/uFEB/Yu4wuSUrb1hhAzX7uBFsEhh3AACFEghDCB7geaDa6SAgxGngFLSgUNNkfIoTwNb4OByYATTutu61Xth7jy72nWXrFYCYOsGJB9KaKjYFB1Ris0mTAxZXAUOAGIcTQFucMAB4EJkgphwF/sntBXVFtGejrbbuyoKoxdJjVTUlSSp0Q4nfAt4An8IaUcr8Q4gkgTUq5DvgnEAB8bEwNfVJKOQcYArwihDCgBanlLUYzdUtrd+ey/OtDzBoRxV2TbdhRXJKlDQPs1bf9c5W2NA64ABBCmAZcNH3v3gG8KKUsBWj6gUhpQ+Os5y6oMUipViy0kC36GJBSrgfWt9j3SJPXl7dy3TYg0RZlcBfbsoq4/+M9pCSE8q+FIy1fY8ESJceMQ1WtGO6qgPkBFyktzhkIIIT4Ge0D02NSym9a3kgIsQRYAhAXp7Ld2nRym0lAJOjrtP61Hr1sd183pmY+O5H9eeXc+U46CeE9efXmZHy9bDCOu6niLNWMZBuWDLjwAgYAU4AbgJVCiPP+KkkpX5VSJkspkyMibNRk6MoqbZgOw0TNfu4wFRicREZuOTeu/JVAPy/evHUcwf7etn2AlFqNQXU824IlAy5ygLVSygYp5XHgMFqgUNrSJTUGNfu5o1RgcAKmoNDTx4vVSy4ippcVC6C35mwR1FWoGoNttDvgAvgcmAqNAysGAsfsWkpXVHUGvPzAL9h291SznzvMJn0MSuelHi/hjnfSCPD1YvWS8bYZlmqOGqpqMxYOuPgWmC6EOADogaVSymLHldp6DQ0N5OTkUFtb23UPCZsB06bCoUO2u6fBAFd8BPoQOHjQdvd1Yn5+fsTGxuLt3bmWBxUYHGjt7lyWfryX2NAevH3ruK4LCqA1IwGEqnQYtmDBgAsJ3Gf8cgs5OTkEBgYSHx9v20ERTRVlas2etlyPXEo43QABERDUVlIG9yClpLi4mJycHBISOpcsUzUlOYDBIPnvD5n8cfVuRsX14tO7L+7aoABax7PwhBA1VFXpnNraWsLCwrouKAAYdOBp48+rQmipMfQ6297XSQkhCAsLs6pmp2oMdlZcVcefP97D5sOFzB8dw/JrEm0/+sickizoFaf9gihKJ3VpUADQN4BPgO3v6+EFhgbb39dJWfv/pGoMdvRLVjFX/fcnth0t5sm5w3hu4Uj7BAVQQ1UV5ycNIPUd/vCyePFiDhxoZ16sh3U1hpkzZ1JWVnbe/scee4xnn3220/fNy8vj2muv7fT1XUXVGOygvKaB5V8fYlXqSeLD/Pn0nosZHmPDURftMQ1VjbvIfs9UlI7SGz/Re3Tsz9LKlSvbP8nTCxrOdqJQmvXr17d/UgfpdDqio6P55JNPbH5va6kaQxcyGCSf7cph2nNb+HDHSe6YmMD6P060b1AAOFsI9VWqxqA4N4PxE30rNYazZ89y1VVXMXLkSIYPH86HH34IwJQpU0hLSwPg9ddfZ+DAgUyZMoU77riD3/3udwAs+u1S7v7LE0ydOpV+/fqxZcsWbrvtNoYMGcKiRYsan7Fq1SoSExMZPnw4y5Yta9wfHx9PUVERAE8//TSDBg3i8ssv5/Dhw2bLumjRIu666y4mTpzIwIED+fLLLwF46623WLBgAbNnz2b69OlkZ2czfPjwxmPz5s1j9uzZJCQk8MILL/Dcc88xevRoxo8fT0lJCQBZWVnMmDGDMWPGMHHiRA7ZcgSXkaoxdAEpJT9mFrH860McOF1BYkwwr98ylsRYOwcEE1PyPDUiSbGRx7/Yz4G8Cpvec2hvXx5NEVqzjxnffPMN0dHRfPXVVwCUl5c3O56Xl8eTTz7Jzp07CQwM5NJLL2XkyJHaQSEoLa9g4/ffsu6rr5k9ezY///wzK1euZOzYsezevZvevXuzbNky0tPTCQkJYfr06Xz++efMm3dumZj09HRWr17Nrl270Ol0JCUlMWbMGLPlzc7OZsuWLWRlZTF16lSOHj0KwC+//MLevXsJDQ0lOzu72TUZGRns2rWL2tpaLrzwQp555hl27drFvffeyzvvvMNpCzouAAAgAElEQVSf/vQnlixZwssvv8yAAQP49ddfueeee9i4cWNn/slbpQKDDekNkm8yzvDq1iz25JQTG9KD/1w/itkjovHwcGDyrhIVGBQXIA2AZ6s1hsTERO6//36WLVvGrFmzmDhxYrPjqampTJ48mdDQUAAWLFjAkSNHtIPCg9nTJiEMOhITE4mMjCQxUUvTNmzYMLKzszlx4gRTpkzBlJrkxhtvZOvWrc0Cw48//sj8+fPx99dGEc6ZM6fVH2fhwoV4eHgwYMAA+vXr1/jJftq0aY1lbGnq1KkEBgYSGBhIcHAws2fPbvzZ9+7dS1VVFdu2bWPBggWN19TV1Zm9lzVUYLCB0+U1rEnP4cO0U5wqqaFvmD9PzhvOwuRY+3Uut6U4S2u3VVlVFRt5dPYw29+04rQ287mVPoaBAweSnp7O+vXrefDBB5k+fTqPPNI4dQRt6kgrhAe+Pj5g0OHh4YGv77k1zz08PNDpdHh5Wfbn0NIRPy3PM33fs2fPVq9pWS7T96YyGgwGevXqxe7duy0qQ2epPoZOyq+o5d3tJ7hp5a9MWL6RZ787Qmwvf1bcmMTGP0/h5vF9nSMogHGoal/bjw9XFFsyNGhBoZU/vHl5efj7+3PTTTdx//33s3PnzmbHx40bx5YtWygtLUWn07FmzZomR8W5Z7QiJSWFLVu2UFRUhF6vZ9WqVUyePLnZOZMmTeKzzz6jpqaGyspKvvjii1bv9/HHH2MwGMjKyuLYsWMMGmT9qnRBQUEkJCTw8ccfA1ow3LNnj9X3bUn9pbBQTb2e9BOlbD9WzI9Hi9hzShu6lhDek99OvZAFY/oQF9bFk9Q6q/iY6nhWnJ++odX+BYB9+/axdOlSPDw88Pb2ZsWKFc2Ox8TE8NBDD5GSkkJ0dDRDhw4lONjYryc8zj2jFVFRUfz9739n6tSpSCmZOXMmc+fObXZOUlIS1113HaNGjaJv377nNWc1NWjQICZPnkx+fj4vv/wyfn5+7fwDWOb999/n7rvv5qmnnqKhoYHrr7/+XF+KjYg2q1+W3kSIGcB/0PLGrJRSLm9x3Bd4BxgDFAPXSSmzjcceBG5HyyfzBynlt+09Lzk5WZpGIXSF2gY9h89Usj+vgv155WTkVXAgr5wGvcTTQzAiNpjLh0RyxbBI+kcEdP2kH2tICX+LgaSb4cpnHF0alyCESJdSJtv7uV39vrbWwYMHGTJkSNc9oPAweHhC2IWdvkVVVRUBAQHodDrmz5/Pbbfdxvz587WDp/eCfygEx9qowK1btGgRs2bNcugcBXP/X5a+t62uMTRZ5nAaWqrhHUKIdS1WYrsdKJVSXiiEuB54BrjOuBzi9cAwIBrYIIQYKKXUW1uuttQ26CmsrKOgspb8ijpySqvJLq4mu+gsJ4qrySuvwRQvA/28GBYdxG2XJHBRvzCS40MJ8HWhilZVvjZ+WyXPU5ydvkHLrGqFxx57jA0bNlBbW8v06dObdRzj4dVmjUE5xxZ/4SxZ5nAu8Jjx9SfAC0L7mD0XWC2lrAOOCyGOGu/3S0cLodMb2HAwn4oaHRW1DVTUNFBRqzNuG6io0VFaXU9BZR3lNee/OXr5exMf1pNxCaH0DfNnUGQgw6KD6RPaw7lrBO1pXOdZjUhSnJiUxjxJ1qVsaXMWsqf3ubkSXeytt96yy3O6ii0CgyXLHDaeY0xZXA6EGfdvb3Ftp9Mf3vXeuc4oISDQ14ugHt4E+XkT1MOLfhE9Gd8vjN6BvkQG+RER5EvvQF9ievWgl7+bLnep0m0rrsCgA2SbfQxW8/CChpquu78bsUVgsGSZw9bOseRa7QbtrI3r5enB13+cSKCfFgwCfLwcO3fAWRRnab9swX3aP1dRHKVx1nMXNtN6ekNdZdfd343YYriqpcsc9gEQQngBwUCJhdcClq2NOyQqiNgQf4L8vFVQMCk5BiHxaqiq4twa8yR1cY1B6sHQpV2YbsEWgcGSZQ7XAbcYX18LbDQuZLIOuF4I4SuESEBbEzfVBmVSTEqOqRnPivMzzS/o6hoD2K2fwZVZHRiklDrAtMzhQeAj0zKHQgjTfPHXgTBj5/J9wAPGa/cDH6F1VH8D/LarRyR1K6asqmoOg+LsGmsMrff1lZWV8dJLL3X+GR7eTLn2DtJSf+38PVqwKOW3C7JJeLZgmcNaYEHL64zHngaetkU5lBYqT0NDtaoxKM5P36CtMOjR+mdVU2C45557OveMxhqDbYas6vV6y1J+uyCVEsOdNQ5VVTUGxcnpG8Cz7ZGBDzzwAFlZWYwaNYqlS5dSVVXFZZddRlJSEomJiaxduxbQspoOGTKEO+64g2HDhjF9+nRqamoaA8PHaz5j3LhxDBw4kB9//PG852zevJlJkyYxf/58hg4dyl133YXBYAAgICCARx55hJSUFH755ZdmKb8DAgJYtmwZY8aM4fLLLyc1NZUpU6bQr18/1q3TWtf1ej1Lly5l7NixjBgxgldeecVm/4S2pHok3Zkaqqp0la8fgDP7bHe/hmqIGATzX271lOXLl5ORkdGYQE6n0/HZZ58RFBREUVER48ePb8x2mpmZyapVq3jttddYuHAha9as4aYbb9Sua6gnNTWV9evX8/jjj7Nhw4bznpWamsqBAwfo27cvM2bM4NNPP+Xaa6/l7NmzDB8+nCeeeOK8a86ePcuUKVN45plnmD9/Pg8//DDff/89Bw4c4JZbbmHOnDm8/vrrBAcHs2PHDurq6pgwYQLTp08nISHBFv+KNqMCgzsrOaZ9CrNDCgBFsYo0aE1JHblESh566CG2bt2Kh4cHubm55OfnA5CQkMCoUaMAGDNmjLbugRCAB1dfNa35fjPGjRtHv35aE+wNN9zATz/9xLXXXounpyfXXHON2Wt8fHyYMWMGoKXJ9vX1xdvbm8TExMbnfPfdd+zdu7dx1bby8nIyMzNVYFDsqDhLG6rq4SRZXhX3ceXy9s+xlDTA6T0QeEGHLnv//fcpLCwkPT0db29v4uPjqa2tBZqnr/b09NSakgCEwNfLo3G/Tmd+hFJrKbP9/Pzw9DT/++Tt7d14nrmU2aAFs+eff54rrriiQz+rvak+BndWckw1IynOz8I5DIGBgVRWnpugVl5eTu/evfH29mbTpk2cOHGi/WcJYdFw1dTUVI4fP47BYODDDz/kkksuaf/eFrjiiitYsWIFDQ3az3zkyBHOnu38WtRdRdUY3JXBoAWG/pc6uiSK0jZTYGin8zksLIwJEyYwfPhwrrzySpYtW8bs2bNJTk5m1KhRDB48uP1nCQ8tMLSTVfqiiy7igQceYN++fY0d0bawePFisrOzSUpKQkpJREQEn3/+uU3ubUs2Sbttb86entgplJ2C/xsOs/4Nybc5ujQuRaXdNq/L0m7XlEJpNkQMBu8etr9/U2cLoTwHIoe3mrBv8+bNPPvss3z55ZddW5YuZk3abdWU5K5MI5KsyG2vKHbRWGPownQYJqbmKn191z/LhammJHdVfFTbqsCgODt9g9bE08FRSZ1iaq5qY12GKVOmMGXKlK4vixNTNQZ3VXwMvP0hMMrRJVGUtunrtU/y9lj3xMazn92VCgzuqvioNiLJlRcZUpxOl/RJ6hvs04wEWoZVhNuv5Gbt/5MKDO6q+Khata0LCSFmCCEOCyGOCiEeaOO8a4UQUghh985sW/Pz86O4uNj2wcFgx8AghPYsNw4MUkqKi4vx8+v8Mqmqj8Ed6Rug7AQMm9f+uUqHWbjOOUKIQOAPgO3SeTpQbGwsOTk5FBYW2u6mUkJ5LvgFwhk7ra5WVQgUQUC1fZ7nAH5+fsTGdj7jgQoM7qjspDZWW3U8dxVL1jkHeBL4B3C/fYvXNby9vW2fuqGqAJ69CK78J4xeYtt7t+aj5ZCfAb9Pt8/zXJBqSnJHxSp5Xhczt855s7XKhRCjgT5SStceDN/VKowLNgZF2++ZQTHac11wDpe9qMDgjtRQ1a7W5lrlQggP4N/An9u9kRBLhBBpQog0mzbRuIrGwGDH0XNBUVo219py+z3TxVgVGIQQoUKI74UQmcZtiJlzRgkhfhFC7BdC7BVCXNfk2FtCiONCiN3Gr1HWlEcxKj4Kfr3AP9TRJXFX7a1VHggMBzYLIbKB8cA6cx3Qlqxl7tYqTYEhpu3zbMlUO6kwu7y8gvU1hgeAH6SUA4AfjN+3VA38j5RyGDAD+D8hRK8mx5dKKUcZv3ZbWR4FtFnPYWqoahdqc51zKWW5lDJcShkvpYwHtgNzpJTOm+/CUcpztSGkPe0YFE1BqDzHfs90MdYGhrnA28bXbwPnDYORUh6RUmYaX+cBBUA3/GhkR8VZqhmpC1m4zrliibKT2noh9kwN3ytO25aftN8zXYy1o5IipZSnAaSUp4UQvds6WQgxDvABsprsfloI8QjGGoeUss7KMnVvDTXaJyHV8dyl2lvnvMX+KfYok0sqPwXBfdo/z5YCLtBmWpedav/cbqrdGoMQYoMQIsPM19yOPEgIEQW8C9wqpTQYdz8IDAbGAqHAsjau796ddJYqOQ5Itc6z4hrKTp77BG8vHh5aLaVM1Rha026NQUp5eWvHhBD5QogoY20hCq2ZyNx5QcBXwMNSyu1N7n3a+LJOCPEmbYz3llK+CrwKWnri9srdbakRSYqr0NVD5Rn7BwaAXn202opilrV9DOuAW4yvbwHWtjzB2Dn3GfCOlPLjFseijFuB1j+RYWV5lMZ026rGoDi5ihxA2r8pCSA4TjUltcHawLAcmCaEyERLD7AcQAiRLIRYaTxnITAJWGRmWOr7Qoh9wD4gHHjKyvIoxUehZ2/wDXR0SRSlbaamnF4OCAy94qDqDDTU2v/ZLsCqzmcpZTFwmZn9acBi4+v3gPdauV6tO2lrRZkQPtDRpVCU9pk+sTuixmAKRhW5qnZthpr57E6khMJDEDHI0SVRlPaVn9IW6LHn5DYTUzBSHdBmqcDgTqoKtGn+KjAorqDspLaQlJeP/Z9t6vBWgcEsFRjcSeEhbauakhRXUOaAOQwmQdFabUWNTDJLBQZ3UnRE20YMdmw5FMUSZScd0/EM2mI9QTGqxtAKFRjcSeFh8A2CwAscXRJFaZuuThuuGurAVQZD4o0TQpWWVGBwJ4WHtGYklTxPcXZlJ0EaIMTGC/90RGgClKrAYI4KDO6k6IhqRlJcg+mTukNrDAlwthDqKh1XBielAoO7qCmFqnyIUB3PigsoOaZtQx1ZYzAGJdWcdB4VGNxFobHjOVwNVVVcQOlx8Amw7zoMLZmCkilIKY1UYHAXRYe1rZrDoLiCkuNaU44j+8NM/Ruqn+E8KjC4i8LD4OXnmEyVitJRJccc24wE4BcE/uGqKckMFRjcReEhCBtg35WwFKUzDHooO+H4wABaGVRT0nlUYHAX+fshcqijS6Eo7avIBX29Y4eqmoT2g9JsR5fC6ajA4A7OFkPlaYgc7uiSKEr7GkckOXCoqkloP20pXJV+uxkVGNxBvnF9owtUYFBcQFGmtnWGnF5hFwLy3AJXCqACg3swBYbIRMeWQ1EsUXgYfAKdI3WLKTgVHnZsOZyMVYFBCBEqhPheCJFp3Ia0cp6+yept65rsTxBC/Gq8/kPjMqBKR53J0FZtC3DgmHBFsVTREW0ipjOkbgkfAIhztRgFsL7G8ADwg5RyAPCD8XtzaqSUo4xfc5rsfwb4t/H6UuB2K8vTPeXvU81IiusoOuIczUgA3j20Id5FqsbQlLWBYS7wtvH128A8Sy8UQgjgUuCTzlyvGOkbtGqw6nhWXEFthTZQwsrAUFBRS0ZuOYfOVFBe02BdmcIHnktZrwBWrvkMREopTwNIKU8LIXq3cp6fECIN0AHLpZSfA2FAmZRSZzwnB3DAGn8uruiINvTvAtW/oLgAKzqec0qrefeXE6zPOM2pkppmx4ZFBzF/dAwLx/YhyM+7YzeOGATZP4HBAB6q2xUsCAxCiA2AuV6i/+3Ac+KklHlCiH7ARiHEPqDCzHmyjXIsAZYAxMWp2b2Nzpg6nlWNQXEBjYtJWZ66pbpex39+yOSNn45jkDB1UASLLk4gNqQHOr0kq7CKjYcKeOqrg7y46SjLZgzmurF9EJb2YYQPAF2NtppbSN9O/FDup93AIKW8vLVjQoh8IUSUsbYQBRS0co884/aYEGIzMBpYA/QSQngZaw2xQF4b5XgVeBUgOTm51QDS7eTvA08fYyeaoji5osPg4aUtkmOBI/mV/Pb9nWQWVLFgTCz3ThtIdK8e5533h8sGsOdUGU9/dZAHPt3HhoMFPLtgBL38LRjPYko8WXREBQYja+tN64BbjK9vAda2PEEIESKE8DW+DgcmAAeklBLYBFzb1vVKO85kaGsweHaw+qwojlBwCEL7W/R+3ZZVxNUvbaO0up73bk/hnwtGmg0KJiP79OLDO8fz6OyhbDlSwNUrtpFbVtPq+Y1MtZeCg5b+FG7P2sCwHJgmhMgEphm/RwiRLIRYaTxnCJAmhNiDFgiWSykPGI8tA+4TQhxF63N43crydC9Swpm9cMEIR5dEUSyTn2HRCLpvMk6z6I0dRAX78cXvL+GSAeEW3V4Iwa0TEnjv9hQKK+u4dsU2MvPbWYjHPxQCo8/NB1KsCwxSymIp5WVSygHGbYlxf5qUcrHx9TYpZaKUcqRx+3qT649JKcdJKS+UUi6QUtZZ9+N0M2UnoLoYYpIcXRJFaV9NqdaO305/2MZD+fz2g10Miwni47suIiq49VpCa1L6hfHRnRehM0h+s/JXThZXt33BBcPP9dcpauazS8vdqW1jxji2HIpiifz92raNEXSpx0u4+72dDI0K4p3bxlnWR9CKIVFBfLA4hQa9gZte/5WCyjbyIUUO1/o/dOqzKajA4Npy08HTFyKHObok3Y4QYoYQ4rAQ4qgQ4ryJnUKI+4QQB4QQe4UQPwghVK9mOyPoThZXs/jtHcT06sFbt44lsKPDTs0YEBnIm4vGUlRVx61v7qCmXm/+xAuGg0GnUmMYqcDgyvJ2QdQI1fFsZ0IIT+BF4EpgKHCDEKJlzvNdQLKUcgTaJM5/2LeUTih/H/iHmc2RVNug554P0pHAW7eOIyzA12aPHR0Xwgu/Gc2B0xX8Zc1etHEvLZjyjKl+BkAFBtdl0EPebohW/QsOMA44auwjqwdWo2UBaCSl3CSlNDVsb0cbjt29ncnQagtm5hc8/dVBMnIr+NeCkcSF+dv80ZcOjmTpFYP4Yk8eL28xszBPWH/w6qH6GYxUYHBVhYeg4azqeHaMGOBUk+/bm7V/O/C1uQNCiCVCiDQhRFphYaENi+hk9DrtPWumf2Hdnjze3X6CJZP6MX1Y12VcvXtyf2aNiOIf3x5i0+EWU648PKH3EG2Un6ICg8s69au27TPOseXonsxNqTU76VIIcROQDPzT3HEp5atSymQpZXJEhBtnxy04ALpaiBrVbHdWYRUPrtnLmL4hLL3C8tnQnSGE4J/XjmTwBUHc++Fu8lrOcYgeBaf3aKkxujkVGFzViV8gINI5lkfsfnKAPk2+NztrXwhxOVrqmDndfih2bpq2jT03gq6mXs897+3E19uTF34zGm/Prv9z1MPHkxd/M5oGnYE/rNqFTt8kCMQkQ10FFKsU3CowuKqT2yFuvHPktO9+dgADjOuJ+ADXo2UBaCSEGA28ghYUzKaK6VZy0rWO5yYfZB5Zm8GRgkr+77pRnZqr0Fn9IgL429WJpJ0o5bnvm2RVjU02lnWH3crirFRgcEXlOVB+EuIudnRJuiVjbq/fAd8CB4GPpJT7hRBPCCFM6438EwgAPm65QFW3lJumfSI3fpD5KO0UH6fn8PupFzJpoP2b0OaOiuH6sX14aXMWW44Y+3bCBoBvMOSk2b08zsbatNuKI5zcrm3jxju2HN2YlHI9sL7FvkeavG41+WS3U1uhzQ8Yfg0Ah85U8MjaDC7uH8YfL3fcgj2Pzh7GrpNl3Pfhbtb/cSKRQX4QM/pcs1c3pmoMrujENvAJUKm2FdeQtxOQEDOGqjod97y/k0A/b/5z/Wg8PRzXFNrDx5MXbxxNdb3+XH9DTDLkH4D6dlJouDkVGFzRsc3QdwJ4qgqf4gJOpQICGZ3Eg5/uI7voLM/fMJqIQNtNYuusC3sH8tS84fx6vIT//pAJsWNB6rWsAt2YCgyupuwklGRBvymOLomiWOb4Vrggkff2VvDFnjz+PH0Q4/uFObpUja4ZE8u1Y2J5ftNRftENAuEB2T86ulgOpQKDq8napG37T3VsORTFEg21cCqVovBxPPnFAaYOiuDuyf0dXarzPDF3GP0jAvj9Z1k09B4Bx1VgUFzJsU0QGKUtzqMozi5nB+jr+OeR3oQH+PDcwlF4OLBfoTX+Pl68+Jskquoa+PbshcjctG7dz6ACgysx6OHYFq0ZSc1fUFyAPL4VAx58V9WPF25MIqRn59Nod7VBFwTy+JxhfFKSgNDXQ06qo4vkMFYFBiFEqBDieyFEpnEbYuacqcZx3KavWiHEPOOxt4QQx5scG3X+U5RGp1KhpgQuVCMhFddwZs93ZBj68ruZySTFnffnweksTO5D5LCp6KQHOTu/dXRxHMbaGsMDwA9SygHAD8bvmzFmmRwlpRwFXApUA981OWWp6biUcreV5XFvh78CD28YMN3RJVGUdm3POEpE2V5Oh1/MbRPiHV0ciwgh+Ou1KRzwHMzZjK8prOyemUysDQxzgbeNr98G5rVz/rXA103SESuWkhIOfQUJk8AvyNGlUZQ25ZRWs3bN23gJA5Nm3YxwoabPAF8vIsfOZxDHefjtb6jXdb+ketYGhkgp5WkA47Z3O+dfD6xqse9p4ypX/xZCtDqwudukJ25N4WEoOQaDZzq6JIrSptoGPXe/t5OJhjT0PcLpEZ/i6CJ1WGSytrxGxOlNPLouw/ziPm6s3cAghNgghMgw8zW3vWtb3CcKSETLL2PyIDAYGAuEAstau77bpCduzYG1gIBBKjAozktKyUOf7uNgbjHTfPbhOWgGeLjgGJfwgRCSwK0Rh1iVeor3tp9wdInsqt2ps23lfBFC5AshoqSUp41/+NvKIrkQ+ExK2dDk3qeNL+uEEG8C91tY7u5FSti7GuIvgaBoR5dGUVr1/MajfLorl3+PqcB7fyUMmuHoInWO0D6E9dvxGrMG+PP4Fwe4sHcgF/V3nol5XcnaUL4OuMX4+hZgbRvn3kCLZiRjMEFoDZDzALWunjm56Voz0ojrHF0SRWnV57tyee77I1w9OoZ5Xj9rmUovnOboYnXe8GsQ+nqeHZ5NfHhP7nw3jSP5lY4ulV1YGxiWA9OEEJnANOP3CCGShRArTScJIeLRFjbZ0uL694UQ+4B9QDjwlJXlcU97VoOXHwyd0/65iuIAv2QV85dP9pKSEMrf5/RHHPxCe796+zm6aJ0XkwSh/fE7uIa3bh2Ln7cnt7yRev7Kb27IqsAgpSyWUl4mpRxg3JYY96dJKRc3OS9bShkjpTS0uP5SKWWilHK4lPImKWWVNeVxS/VnYd9HMPgq8At2dGkU5Ty7Tpay+O0d9A3z55Wbx+B79Fuor3L9Gq4Q2s+Q/ROxHiW8fds4qmp1/M8bqZRV1zu6dF3KBXuFupm9H0FtOYy9w9ElUZTzHDxdwaI3dxAe6Mt7i1Po5e8Du96FoFgtA7CrG7EAkLDrPYZEBfHaLcmcLKnm5tdTKa9uaPdyV6UCgzOTElJfgwsS1aI8itM5dKaCm1//FX8fT967PUVb6KbgkJYWfuztrjkaqaXQflo/SdoboKtnfL8wXrlpDIfPVHLj69vdtubgBv9zbixrIxTsh3F3qtxIilPZebKU617ZjpeHB+8vTqFPqL92IPUV8PSFpFvavoErSbkTqvLhoLY669TBvXn1f8ZwJL+K37z2K8VV7jc7WgUGZyUlbHlGq5KPWOjo0ihKo58yi7hp5a+E+Hvz8V0X0S8iQDtQVagNlEhcAD3daFhn/8sgtD9s+6/2ewlMGdSb1/4nmazCKq5ZsY3sorMOLqRtqcDgrLI2wqlfYeJ94OX4la4UBeD9X0+w6M1U4kL9+eiui87VFAB++jfoauGSex1XwK7g4QET/wyn98ChLxt3Tx4YwQd3jKeiVsfVK7aRfqLUgYW0LRUYnJFeB9/9FYLjYPRNji6NotCgN/Dw5/v4388yuGRAOB/ddRG9A5sMRa3Igx0rYeRvIPxCxxW0q4y4DsIGwMantfT3RmP6hvDp3RcT5OfFDa9tZ3XqSbdIn6ECgzNKe13rW7jiaVVbUBzuVEk117+6nfe2n+TOyf14/ZaxBPl5Nz/pu4cBCZOXOqSMXc7TCy79Xyg8qHVENxEf3pNP75lASkIoD3y6jz9/vIfqep2DCmobKjA4m7KTsPEp6DcVhsx2dGmUbkxKyac7c7jyPz9y5Ewl/71hNA9eOQTPliuwHf0BMtZozS0h8Q4pq10MnactkvXDE1CZ3+xQaE8f3rp1HH+6fACf7cpl9vM/sfOk6zYtqcDgTAx6+PROrYNr9v+pkUiKw5wqqeaOd9K576M9DI0KYv0fJzJnpJk8XdUl8MUfIexC9+tbaEkIuOo50NXBut+BoXk6bk8PwZ8uH8i7t6VQU6/nmhXbePLLA9TU61u5ofNSgcGZ/PA4nNwGVz3r3p+8FKdV26DnhY2ZTPv3FrZlFfHQzMGsWjK+eSezicEAn98DlWfg6te6R7NnWH+tiTfzO22UkhmXDAjn23sncWNKHK//dJxL/7WZz3blYDC4Tt9Du9lVFTtJexN+/g+MXez6qQQUl1On0/PhjlO8uOko+RV1zEy8gIevGkp0rx7mL5ASvv8rHPkarvyHlleouxi7GLJ/gg2PQWgCDD1/BYJAP2+empfInJExPPnlAe79cA9v/pzNfdMGMnlghNMvXKQCgzNIfxu+vFdbsnPGM6oJSbGb8uoGVu84ydvbstV4L+sAAAvzSURBVMkrr2VsfAj/uX404/u1MQ9BSti8HH55AcYt0b66EyFg3ktQkQtrFoNXDxhofrndcQmhrP3tBD7fncuz3x5m0Zs7GBoVxN1T+jNj+AV4ezpno40KDI5kMMCmp+DHf2nT7he+q41+UJQuZDBI0k6U8tmuHD7flUdNg17LinrNCCYNCG/706yuHr55QBs5N/I33feDjE9P+M1H8M5cWHU9zPo3jDE/29vDQ3B1UiyzRkTz+e5cXt6cxe9X7SIi0JeFybFclxxHXJiZpjoHUn+FHKU0G9b+DrJ/hNE3w1X/6h5ttIpD6A2SPTllfH8gn3W788gtq6GHtyezRkSxaEI8w6ItyNxblAmf3amtDzLhj3DZY+6RD6mz/ENh0Vfw0f/AF3+AE9tgxt+1/Wb4eHmwMLkP1yTFsvFQAatTT7JicxYvbsoiMSaYKxMv4MrhUSSE97TzD3I+4YqTMZKTk2VaWpqji9E51SXw03NacjwPb5jxNy0wdMdPXU5KCJEupUy293Nt+b42GCTHiqrYeaKMH48W8WNmIWXVDXh6CCYOCGfeqBimDY2kp68Fnw2rCmDb8/Dry+DdA+Y8b7Zdvdsy6GHLP2DrP8E3EC59WPudtmAtiryyGtbtyePrjDPsOVUGQHyYPxdfGM6E/uGk9AslPMB2HxgtfW9bFRiEEAuAx4AhwDgppdl3tRBiBvAfwBNYKaU0LeiTAKxGW+95J3CzlLLddIUuFxgMBshNg53vQMan0FANI6+Hqf8Lvfo4unRKC5b88rT2nm5y3Bd4BxgDFAPXSSmz27pnZ9/XtQ16jhedJbOgiqP5lezOKWf3yVIqarVJVuEBPkwaGMGUQb2ZNCBcS43dHim1lCx7Vmn5j/T12qCIaU9AQO8Ol7FbyD8AX/9FawXoGQHJt0HiQotngueW1fDd/jP8lFnEr8dLqKrT/v9ievVgeEwQI2J7MSw6iP4RAUT36nH+fBIL2CswDAEMwCvA/eYCgxDCEziCtsJbDrADuEFKeUAI8RHwqZRytRDiZWCPlHJFe891+sCg10HBAa3KfSoVjm6AswXg7Q/Dr4Hx90DkUEeXUmlFe788bb2nm5xzDzBCSnmXEOJ6YL6Uss3hZq29r+t0es6U15JXVsvp8hpOl9eSW1bD6bIasourOVF8FtNISA8BA3oHktS3F6PjQkiKC6FfeE882vsjoqvXlo/NTYPjP8LxrVCZp3WsJl4DE+51z1QXtiYlHNsE21+GzG+1fb2HQcIkiEuB2HHauu3ttBDo9Ab25paTll3CvtwKMnLLOd4kUZ+Ppwd9w/xJCO9JfHhPooL9jF89iA/rSbC/t9n7WhoYrOpjkFIeND6srdPGAUellMeM564G5gohDgKXAr8xnvc2Wu2j3cBgN1Jqn5R0tdBQq211tdon/uoSqCk1bku0tLyl2dpX2UkwGKfE9wjRZjEPvAIGzQS/IEf+RIptmH1PAweanDMX7f0M8AnwghBCyE58Evtoxyn+unZ/s32hPX2ICvZjSFQgs0dGc2HvAAb0DiAhvCd+3p5a80b92f9v7/xi5KrqOP75zs5u2+02bem23dJaaRUtSEKsxrbKQyNomkpoSIzhSRIfCGITDSFo0sQYeJEYfTARSU2NYnjAf0iFEpCI8alVaWgraSVtimHZov0Tt2Ur7s7Oz4dztjsz3Nm5O3Nn9s7u75OczNl7z875zpnvvWfOOffeH4xfgUvvhIhqE1fhvcvhR8qVfwXPXjkX1g4unZn2bP8g3HAbfGQX3HRnmB5x0iHBhz4b0ugwnPw9nHoeXv0ZHImntr6BEOdh1Ydh+fowuphKi1dAXz/F3n62XjfA1qEhKG6CQoHR/05w6txlzl4Y4+zFMc6eH+PshTH+9MZ5xkvTN9s9vOujPLCztU68E4vP64G3Kv4eBrYBq4D/mFmpYvv6pmspT8LjOwADK8dk06+J22Oioty1fZPhDkdSHsdLVoab0tbdGuZf194Sru1eucnXD+Yf9TydWMbMSpJGCZ6/MNvKPsdh9gx+l14ZRZXpoUzBJuF/k/D2JLxVCtOV5VLwbbkUfd2AxStgYG04QW35AqzeAkO3wOqbFvaiclYs3wDbvxpSaRzeOQEjR+Hi6ZBGjoZOYzJFPIdCkeWFItsKRbYVeqBQvJZssIcyPZRMlMowpq8Bbe4YJL0MDCXs2mdmz6aoI+msaDNsr6fjPuA+gI0bNyZXs2YLqBASms5L06+J22v/RyEVF0+n3sr8ktAR9K+CJdeFvF9mupBI491U/m7saxhaOwQbb40ngnhSUKHm756Ynzpp9IZLKvuWhl+off0xvwwGVsPSNakWR52MKPbBhk+EVIlZGM2NnQ/xLN4bhYkxGL8aRngTV0O+XKqbVJ6kZ3KCHoxFVmbp2oRHl8xWbqMCZnZHi3UMA5UrrBuAEcIvpxWSinHUMLW9no79wH4Ic7HvK1AowJeebFGq46SinqeTygxLKgLLgUu1b9TQ1xAe3LZ5Z2uKnXwiham6RcvC9FJO6MR48a/AjZI2SeoD7gEOxrnWV4AvxnL3AmlGII4z1yR6uqbMQYKnIXj8j82sLzjOXNBSxyDpbknDwA7geUkvxu3XSzoEYX4V2Au8CJwEfmlmUytp3wQelHSaMP96oBU9jtMJ6nla0iOS7orFDgCrorcfBL41N2odZ/b4DW6OU8N8uMHNcZJI622/9MBxHMepwjsGx3EcpwrvGBzHcZwqvGNwHMdxqvCOwXEcx6miK69KknQe+Ged3YM08diBNpEXLXnRAd2h5YNmtrrTYtzXTeFa3s9MOlJ5uys7hpmQ9Le5uNQwibxoyYsOcC3NkietriWZvGjJQodPJTmO4zhVeMfgOI7jVDEfO4b9cy2ggrxoyYsOcC3NkietriWZvGhpWce8W2NwHMdxWmM+jhgcx3GcFuj6jkHSdyS9Lem1mHbXKbdL0j8knZaU+ZMuJX1P0ilJxyU9I2lFnXJvSjoRtWb6xLRGn1HSIklPx/1HJN2QZf0V9XxA0iuSTkp6XdLXE8rslDRa8b19ux1aYl0ztrkCP4ztclzS1nZpSUtefB3rcG+zwHxtZl2dCHF1H2pQpgc4A2wG+oBjwM0Z6/g8UIz5x4DH6pR7ExhsQzs0/IzAA8ATMX8P8HSbvpN1wNaYXwa8kaBlJ/BchzwyY5sDu4EXCFHXtgNHOqGrgeZc+DrW4962heXrrh8xpORa8HYzGwemgrdnhpm9ZNPxqw8Tonp1kjSfcQ/w85j/NXC7lH1AajM7Z2ZHY/4KIWZB8/G8288e4EkLHCZEFlw316JS0HZfg3t7ioXk6/nSMeyNQ6WfSlqZsD8peHs7v9CvEHrqJAx4SdKrCvF+syLNZ6wKUA9MBahvG3FI/3HgSMLuHZKOSXpB0sfaKKNRm3faH2nJm6/BvQ3Mf193RQR7SS8DQwm79gE/Bh4lNNKjwPcJ5q16i4T/nfXlWDPpMLNnY5l9QAl4qs7bfMbMRiStAf4g6ZSZ/Xm2WpLkJWxrKkB9VkgaAH4DfMPMLtfsPkq4Pf/dOH/+O+DGNklp1OYdbZdrlebE1420uLdrKloAvu6KjsHM7khTTtJPgOcSdqUJ3t6yDkn3AncCt1uc5Et4j5H4+m9JzxCGyVkcPJkFqM8CSb2Eg+cpM/tt7f7KA8rMDkl6XNKgmWX+rJkUbZ6JP5rQlQtfp9Hi3g4sFF93/VRSzZzZ3cDfE4qlCd7eqo5dhBjWd5nZ1TpllkpaNpUnLOol6W2G3ASoj3O7B4CTZvaDOmWGpuaAJX2K4MWLbdCSps0PAl+OV3FsB0bN7FzWWmZDXnwdtbi3WWC+7sTqeTsT8AvgBHA8NsS6uP164FBFud2EqwjOEIbHWes4TZjPey2mJ2p1EK6qOBbT61nrSPqMwCOEAxpgMfCrqPUvwOY2fSe3EYasxyvaYzdwP3B/LLM3tsExwoLmp9ukJbHNa7QI+FFstxPAJ93X7u2F7Gu/89lxHMepouunkhzHcZxs8Y7BcRzHqcI7BsdxHKcK7xgcx3GcKrxjcBzHcarwjsFxHMepwjsGx3EcpwrvGBzHcZwq/g89JCXf8fxbDgAAAABJRU5ErkJggg==\n",
      "text/plain": [
       "<Figure size 432x288 with 2 Axes>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "import matplotlib.pyplot as plt\n",
    "import numpy as np\n",
    "\n",
    "def sigmoid(x):\n",
    "    return 1 / (1 + np.exp(-x))\n",
    "\n",
    "def sigprime(x):\n",
    "    return sigmoid(x) * (1 - sigmoid(x))\n",
    "\n",
    "def tanh(x):\n",
    "    return 2 * sigmoid(2 * x) - 1\n",
    "\n",
    "def tanhprime(x):\n",
    "    return 4 * sigprime(2*x)\n",
    "\n",
    "X = np.arange(-5.0, 5.0, 0.05)\n",
    "S = np.array([sigmoid(x) for x in X])\n",
    "T = np.array([tanh(x) for x in X])\n",
    "Sp = np.array([sigprime(x) for x in X])\n",
    "Tp = np.array([tanhprime(x) for x in X])\n",
    "\n",
    "plt.subplot(1, 2, 1)  # 1 line, 2 columns, position 1\n",
    "plt.plot(X, S, label=\"sigmoid\")\n",
    "plt.plot(X, T, label=\"tanh\")\n",
    "plt.legend()\n",
    "\n",
    "plt.subplot(1, 2, 2)\n",
    "plt.plot(X, Sp, label=\"sigmoid prime\")\n",
    "plt.plot(X, Tp, label=\"tanh prime\")\n",
    "plt.legend()\n",
    "\n",
    "plt.show()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "The \"*symmetric about the origin*\" argument suggests trying to shift the sigmoid a little down to satisfy this condition: using $\\sigma(z) - \\frac 1 2$ as an activation function (by executing `exec_shifted_sig.py`), we get:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 4,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "image/png": "iVBORw0KGgoAAAANSUhEUgAAAYQAAAD8CAYAAAB3u9PLAAAABHNCSVQICAgIfAhkiAAAAAlwSFlzAAALEgAACxIB0t1+/AAAADl0RVh0U29mdHdhcmUAbWF0cGxvdGxpYiB2ZXJzaW9uIDIuMi4yLCBodHRwOi8vbWF0cGxvdGxpYi5vcmcvhp/UCwAAIABJREFUeJzt3Xl4VdW98PHv7ww5mUNmSAIEFAQUjIKMKkirL4441dZWW29rbWvbe6/3YhV92mu11NrSentr61sVp16LWlHRvjgi4gAoAZlnlSEJZCJzck5yzlnvH3snnAyQk4mQnN/nec6Ts/dee5+1TpL922utvdYWYwxKKaWUo78zoJRS6tSgAUEppRSgAUEppZRNA4JSSilAA4JSSimbBgSllFJAmAFBROaJyG4R2Scid3ewfaSIrBSRLSLyvojkhGx7SES22a+vh6wfJSKfiMheEXlBRKJ6p0hKKaW6o9OAICJO4M/ApcAE4EYRmdAm2WLgWWPMJOB+4EF738uBc4E8YBpwp4gk2vs8BDxsjBkDVADf63lxlFJKdVc4NYSpwD5jzBfGmEbgeWB+mzQTgJX2+1Uh2ycAq40xfmNMHbAZmCciAswFXrLTPQNc3f1iKKWU6ilXGGmygUMhywVYV/uhNgPXAX8ErgESRCTVXv9fIvIHIBa4CNgBpAKVxhh/yDGzO/pwEbkNuA0gLi5u8rhx48LIslJKqWYbNmwoM8akd5YunIAgHaxrO9/FAuAREbkF+AAoBPzGmLdF5DxgDVAKrAX8YR7TWmnMY8BjAFOmTDH5+flhZFkppVQzETkQTrpwmowKgOEhyzlAUWgCY0yRMeZaY8w5wL32uir75yJjTJ4x5mKsQLAXKAOGiIjreMdUSil1coUTENYDY+y7gqKAbwCvhSYQkTQRaT7WQuBJe73TbjpCRCYBk4C3jTWj3irgenuf7wDLe1oYpZRS3ddpQLDb+X8CvAXsBF40xmwXkftF5Co72Rxgt4jsATKBRfZ6N/ChiOzAava5KaTf4C7gP0RkH1afwpJeKpNSSqlukIE0/bX2ISilVNeJyAZjzJTO0ulIZaWUUoAGBKWUUjYNCEoppQANCEoppWwaEJRSSgEaEJRSStk0ICillALCm8tIqS5pCjbxwq4XqPfXMyt7FuNTxuMQvfZQ6lSnAUH1qv1V+1n44UK2lW8D4E+f/YmU6BRmZs1kVvYsZmbNJCU6pZ9zqZTqiAYE1SuMMby09yV+t/53RDmj+P3s3zM5czJritbwcdHHfFz4Mf/84p8IwvjU8czKmsX52eczKX0SLof+GSp1KtCpK1SPlTeUc9+a+3i/4H1mDJvBA7MeIDMus1WaoAmyo3wHHxd+zMdFH7O5dDNBEyTBncA5meeQm5jLiIQRDE8czoiEEQyLG4bT4exSPkrrS1lTtIa1h9ciSKvjjUgYQZInCevZTCoSGWOo9FVysOYgB6sPcqjmEAdrDlLprWyX1u10kxOfw4hE629neMJwhsUPw+1w90POey7cqSs0IKiwBYIBNpZs5MuqLzlYfZCDNfY/VfVBAO6YfAffHP/NsPoLqnxVfHL4Ez4u+pgtpVs4VHMIX8B3wn0c4mBY3DCGJ9gn+cQRZMZlsqt8Fx8Xfcyuo7sASIlOweP0cKTuCCbkMRsJ7gSGJw5nZMLIlkBxevLpnfZx1DXVsaN8B03BpnC+pg5lxWUxMnFkrwQkX8DH5pLN7K/e3+r3UFBTQEJUgvX9NJ/IEoeTGJXY7hjVvup2J8ayhrIe520gEYRhccNIjUlF2jyipd5fT2FtIQ3+hm4dOy0mrSWQNP8uEj0d/B4aqzlUfajV76K0obTDYy6/ejmjk0Z3Kz8aEFSvqfRW8sq+V3hh9wsU1hYC4HF6Wp2Y5582n9OTT+/2ZwRNkNL60paTW3FdMUGCrdL4g36KaotaTmBVvioAXOIiLyOPWdlWM9TY5LE4xIEv4KOwppBDNYc4UH2AQzWHWvYtqi0iYAIAJHuSmZE1g/Ozz2dG1gxSo1PZU7GnpalrY8lG/EF/uzx3VXZ8Nudnn8/MrJlMGzaNOHccvoCPgpqClhN7dWM1OfE5LSeS9BjrIVf7q/e31K7yj+TjDXiBY7+H4QnDyY7Ppq6pzvoOqw9R0lDSaZ4yYjJagmN6bPqg7/xPjEpsCZQ58TlEOaOOm9YYQ7m3vOV3c7juMEETPG76Zq3+lnvx93DjuBu73f+mAUG1s7V0K8/vfp6dR3e22+ZxeMhJyGl1ReMQBy/vfZkVX67AF/AxJXMKXz/j6+Rl5JERm9HvJ48qXxVFtUUMTxhOfFR8l/ZtCjZxuPYwW8u2tpxoj3qPApDkSWoJNmOTxzIrexZTh04lzh3XrXwaY9hbsZePij7ik8Of0OBvwOVwkRKdQml9aatajCCtlqOd0cRHxbdcvecm5rZ0zo9NHnvC30N9Uz2Hag5R769vty3OHUdOfA6x7thulUmFr76pnoLaAuqa6tpti3XFMjxheMvvoSkQpKK+scPjpMRG4XJ2739OA4ICrOaFt/a/xdKdS9lWvo1YVyxTh05t1z7f/EcbeuUMEOOK4crRV/KNcd9gTPKYk539kyZoguw6uouPCz/mQPUBJmdOZlb2LDJiM3r1c5oCTXxW8hkfFX1EeUM5OQk5LX0cIxJHEOeO40jdkZary4M1BznqPUpeeh4zs2cyPGF45x+iThn1jX4OlNdzoLyOWl+g3faqhiYOlNex305TUNFAINjxOfnd/5jN6Rldu/BppgEhwhlj+NuOv/HE1ieo8FUwKmkUN467kStHX3nCq+nmK+eDNQep9lVzfs75HbZBKzUQNQWCFFQ0sL+8jgNl1on4eFfk3RU0UFzlZX95HSU1J+4XA0iIdpGbGsfI1FhyU+PITPR02Nd05aQskmK716kdbkDQ+/0GIWMMf9z4R5ZsW8KsrFncctYtTBs6LawOTbfDbTUZJY44CTntA34fVB6ExmPVc3/QUFrjIy4xmcShp4Gzj/7sm7zWZze1b6LpUFQ8DBkBro7bsY0xVNY3UVTVQLjXbekJHjISOj6hnEw+f4CCigbr6rfMuvo9WFZDQ9kBpIO7ejwuB0OToskaEsOwpBiykqJJT/DgcLQuh0OE9HgPCdHH/x36/EFKarwUVXopqmzgcJWXoqoGjlR5KanxtboCj3E7GBIbRW9/W6Piopg5PJphQ2IYlhTNsKSYDvMcE+UkMdoV0qndYL864M7Eeghl39GAMMgYY/jDhj/w9PanuWHsDdw7/d5+b+vviZqqoxTv30FV4W6ayg9g2nTuigkS7yshqeEQSQ2HSPAdQWh99nQBw+z3AZz4E4fjyTgdUkZDfCZ09+QZDEBVARz9Ao5+CdWFQNdq3EEc1EYPpSp6OFUxOVS40qloCFBZ30hFfSO+ps47MdtyOYXk2CiGxLpJjo3C4+77378xUOP12/luosbbhDGGdKlilBxhrqOEHCnBRftmEwCagDL71UMeYLj9aqej86m355/Z4THLe/mYP14P6WN7+aCtaUAYRIwx/Hb9b/nfnf/LjeNuZOHUheFfKQaD4AjjxNHkhcoD9knQflXst67MeyhoTMtJxeetI63pMClUk9DJfuUmgYMmk3wzmgNmBgeCmTS5E8hItK6WMxI8ZMR7KCs7Qsn+nWRVHGZc3ZeM3L8Ot7+2R3muciRR7MziiOsMjiRcRLFzGF5HTKs0xkCt109JrQ9/4FjASKSOkY4SRtYdIbe+mBGynbOkTX66e0HotV9Hu7l/T9lnloA7HpM8CmfaNCRltBWEY1PgBNfk9U1+Smt8VNS1v83XHzSU1/ooqfFRWuOjuMbL0bpG4j0uMhI8pCdEk5EQZdWUEqPJTIgmzuNsd1vpgJQ4rPM0PaQBYZAwxvDgpw+ydNdSbhp/Ez8772etg4ExUF9+7Gr26OetT+reaqv5ovmfNmU0xGdYV70t6b60rohDr4I9SZCSazV/dJE/EMTrD1Ln81PZ0ER1QxNBYxDAEx3LvpTZBIeMwpN5OknZ40gfPoYoT3Trg4gQ74xiAjAhZHWU09FhMKz1+XllYwE/XHuAfSU1xDkD5CTHMjI1lhEpsYxMjWNIjJuCygYOltdx4Gg9B8vr27UFD4lxMzwlFmdUdLvP6EhSsptRacfaiUemxpIW72lXOQkEm3A6eu/kFQwamoJdr2V0lSBEudpfUDidUV2ugcUCI+1XOIJB065pSXWPdioPAkfqjvCnz/7Ea5+/xnfGXM9/Zn0Vqfiy9Qn/6Jfgqz62kzggKefYyT86CSpCrvxD08amHkuXPApSTzu2HJN83H94YwxH6xpb7qBo+7Oy/tgVYE5yDLPHpjN7bDozT08j3tO31yrGGNZ+Xs7qvaUcLK9vyVN947EmjfQED7mpsYxIsU/kaXHkpsYyMiWu2517SvUHvctoMNj2Mnz4B+hgtKTBsN5peD4qwHsu687179XU89PysmOVY3FaV/2hJ/Dmk3rySHB52h23sr6RD/eUsn7nPo4eOUT2iNOYduZopo9OJTaq85O0PxDknR3FPPfJQTYfqqTGd6zNXwSykmLaXSmPyUwgNzW23ztCjTGU1lpNFTnJMcT1cVBS6mTRgDCQeatgxc9gy/OQObFVR1LQGJb5S3muqZjPjZcknFznTucGVwbZic1NPqOsn0nDwXniK9lA0LCloJLVe0pZvaeUzYcqCRpIinFzxtAEthZU0dAUIMrp4LxRyVw4Jp1xwxLJTY0le0hMy0CZslofz396kOc+OcjhKi/ZQ2L4yvgMclPjyE2zmmJykmPwuLo2P5FSquf0ttOB6sAaePkHVtv97LvhwjtbbpOsb6rn3o/u5d2D+YxPGc8D47/JvNx5RLvCa8duVlLtZfWeUj7YW8aHe0uprG9CBM7OGcJP545h9hnpnJ0zBKdD8PkD5O+vsALG7lIefGNXy3FcDiEnOYaMxGg2HaykMRDkgjFp3D//LOaOy+jVtnClVN/TGsKpItAEq34NHz0Myblw7eMw/LyWzSX1Jfz0vZ+ys3wnC6Ys4OYJN3e5iaWhMcDPlm3h9c1FAKTFe6x2+zPSOf/0NFLijj+vS7OyWh9fltWxv6yO/XZ/QEFFA3k5Sdw8I7fbIymVUn1HawgDiTHw2r/C5r/Dud+G//MgeI6dWHeU7+Cn7/2U2sZa/mfu/zBn+Jwuf0RxtZfvP5vP1sIqfjj7NK48exjjhyZ2+e6MtHgPafEezsvVh9woNdhoQDgVrH/CCgaz74aLFrbatPLgShZ+uJAhniE8e+mznJFyRpcPv62wilufyafa28RjN0/h4gmZne+klIo4YQ1hFJF5IrJbRPaJyN0dbB8pIitFZIuIvC8iOSHbfisi20Vkp4j8j9jtHHa63SKyyX717ixiA8XBdfDm3TB2Hsy+q9Wm1z9/nTtW3cGYIWP4++V/71YweHv7Eb72f9fiEHjphzM1GCiljqvTGoKIOIE/AxcDBcB6EXnNGLMjJNli4FljzDMiMhd4ELhZRGYCs4BJdrqPgNnA+/byt4wxg7RTIAw1R+DFb1u3hl7z11YjhdcUreEXH/+CqUOn8shXHulyx7G3KcBfV3/Bf6/cw6ScITz+7clkJHTtGEqpyBJOk9FUYJ8x5gsAEXkemA+EBoQJwB32+1XAq/Z7A0QDUVhj1d1Acc+zPQj4G61g4KuBm1+FmCEtm3aW7+SOVXcweshoHr7oYaJd0dT6/KzZV8bqPaVsPFjJ1NzkDjtxjTH8c8thHnpzFwUVDczPy+Kh6yYR7dbbPZVSJxZOQMgGDoUsFwDT2qTZDFwH/BG4BkgQkVRjzFoRWQUcxgoIjxhjQp/O8pSIBIBlwK/MQLrlqafeugcOfQLXPwWZxyZdKKwt5PaVt5PoSeS/Zz/C0nUlvLdrG/n7K/AHDXFRTs7MTmLpp4d4Zu0Bzj89jW/PGMlXxmeyuaCSX/1zBxsPVjJuaALP3TqNWaen9WMhlVIDSTgBoaPbUNqeuBcAj4jILcAHQCHgF5HTgfFAc5/COyJyoTHmA6zmokIRScAKCDcDz7b7cJHbgNsARowYoFMyt7XlRVj/OMz8KZx1bcvqSm8lP3znh/gCPv761cd5YHkhb+8oZvywRG69YDSzx6YzeWQyUS4HZbU+Xlh/iP9dd4Db/raBtPgoymobSU/w8NvrJnHd5BwdB6CU6pJOxyGIyAzgPmPM/7GXFwIYYx48Tvp4YJcxJkdE7gSijTEP2Nt+AXiNMb9ts88twBRjzE9OlJdBMQ6h/ij8aTKkjYFbVoDTRVOgiYLaAn7+8c/ZWb6Tv178V5avi+Zv6w7wX1dO4F9mjTru4fyBIO/uLObljYWMG5bIDy4crVMuKKVa6c1xCOuBMSIyCuvK/xvAN9t8WBpw1BgTBBYCT9qbDgLfF5EHsWoas4H/FhEXMMQYUyYibuAK4N3wijbArVrEWhpYOepMDq68vdXDuwXh93N+z6e7hvC3dbv5wYWjTxgMAFxOB/POGsa8s/p+alyl1ODWaUAwxvhF5CfAW4ATeNIYs11E7gfyjTGvAXOAB0XEYDUZ/dje/SVgLrAVq5npTWPM6yISB7xlBwMnVjB4vHeLdur5fO8KFh9czkdD04k/so7cxFwmpU/iitFXMCJxBBNSJrD5i2h+++Zmrjo7i7vmjevvLCulIohOXXESlDeU8+imv/DS7heJNYYfnPMTbpz4XaKcraeK+GBPKd99ej1TR6Xw1L+cpxPBKaV6hU5dcYr45xf/ZNG6RTQ01XFDdQ0/mn43yXk/bJdu1a4SfvL3jZyeEc//vXmyBgOl1EmnAaEPef1efv3Jr8lNGM6ifZsZHTcKptzWKs2uI9Us+n87+XBvGaPT43jmu1NJjNaHryilTj4NCH3onQPvUNNYw7+bIYyuKoYb/t4yGrm0xscf3tnDC+sPEu9x8fMrJnDz9JEdPoZQKaVOBg0IfWjZ3mWMiB3KeZuWwbk3Q/ZkAJZvKuSel7fi8wf5zsxc/nXuGJLDmHpaKaX6kgaEPvJF1RdsKN7Av3udOKLi4Cv/BcCmQ5Xc+Y8tTMxJ4nfXT2J0uj4/QCl1atCA0BdKdvLymz/AZQzzfUG44VmIS6Os1seP/ncDGYkenvj2FK0VKKVOKRoQelNtKbz/axo3PM1rw7OYk3gaaTf9A1we/IEgP35uI0frGln2o5kaDJRSpxwNCL2luggeuwjqy3hv4hVU1Gzi+hl3gcsDwINv7OKTL4/y8NfP5qzspH7OrFJKtae3tPQGv+/YVNbff49l8dFkxWUxI2sGYHUiL/noS26Zmcs15+R0cjCllOofGhB6w5t3Q8F6uPovHIpLYd3hdVwz5hoc4mDn4WruWraFqbkp3Hv5+P7OqVJKHZcGhJ7a+DfIfxJm/RuceTUv73sZhzi4+vSr+bKsju8+vZ6kGDePfOsc3E79upVSpy7tQ+iJwo3w//4TRs+Bub+gKdjEq/te5cLsC6mrj+fGx9biDxqeu3WaPr5SKXXK04DQXXVl8MLNEJ8B1z0JThcfHFxJWUMZMzIu4+t/XQcYnr9tOmMzE/o7t0op1SkNCN0RDMJL/wJ1pfC9tyAuFYBle5aR4knj98sFh8DS70/n9AwNBkqpgUEDQnccWgdffgCX/g6yzgHg88rP+bDwQxxVlxDtcPH370/nNB2FrJQaQDQgdMeWF8EdC3nHHhz3+JYlEIwipuFCnr9tBrlpcf2YQaWU6joNCF3lb4Qdr8K4y8Fj1QCKaot448sVNFZM509XazBQSg1Meh9kV+17FxoqYOINLaue3v40QQMZ5hIuHJPej5lTSqnu04DQVVtegNhUOO0iAMoaynhpzzIaq87hO1PzcDiknzOolFLdowGhK7zVsOdNOPNacFpPNXtu53M0BZuQqov42uTh/ZxBpZTqPg0IXbHzdfB7YZLVXFTTWMPSnc8TrJnI1WfmkRSrj75USg1cGhC6YuuLkJwLOecB8MLuF6jz19JQOpubZ4zs37wppVQPaUAIV80Ra+zBxK+BCA3+Bp7d/ixu33jOGXoWZ2bplNZKqYFNA0K4ti0DE2y5u+iVva9Q4aug6vAFfFtrB0qpQUDHIYRrywswLA/Sx+IP+nl6+9PEm9OJco7j0rOG9XfulFKqx7SGEI7SPXB4c0tn8pqiNRyuO0xp4XS+OXUEUS79GpVSA5+eycKx9UUQB5x1HQCv7nsVjyRi6ibwzWkj+jlzSinVO7TJqK2GCgj4Q1YY2PoPGHUhJAyl0lvJ+4fex181g4vHZzEsKabfsqqUUr0prIAgIvOAPwJO4AljzG/abB8JPAmkA0eBm4wxBfa23wKXY9VG3gH+zRhjRGQy8DQQA6xoXt8bheq23W/A0m90vO3CnwHwxv43aAo2UVeWx/w5WScxc0op1bc6DQgi4gT+DFwMFADrReQ1Y8yOkGSLgWeNMc+IyFzgQeBmEZkJzAIm2ek+AmYD7wOPArcB67ACwjzgjd4oVLeV2EW69LdWE1Ezd0xL/8HyfctJixpFjS+LKbkp/ZBJpZTqG+HUEKYC+4wxXwCIyPPAfCA0IEwA7rDfrwJetd8bIBqIAgRwA8UiMgxINMastY/5LHA1/R0QakvAkwTTftDh5r0Ve9levp1cbmRkaizpCZ6TnEGllOo74XQqZwOHQpYL7HWhNgPX2e+vARJEJNU+4a8CDtuvt4wxO+39Czo5JgAicpuI5ItIfmlpaRjZ7YHaYog//mylr33+Gi5xUVgwnskjk/s2L0opdZKFExA6mr6zbVv/AmC2iHyG1SRUCPhF5HRgPJCDdcKfKyIXhnlMa6UxjxljphhjpqSn9/HU0rUlEJ/Z4SZ/0M/rn7/OlIxZHK2J0oCglBp0wgkIBUDoNJ45QFFoAmNMkTHmWmPMOcC99roqrNrCOmNMrTGmFqtJaLp9zJwTHbNf1BZDfEaHm9YUraHcW85Iz2wApozU/gOl1OASTkBYD4wRkVEiEgV8A3gtNIGIpIm09MIuxLrjCOAgVs3BJSJurNrDTmPMYaBGRKaLiADfBpb3Qnl65gQ1hFf3vUpKdAo1R08jMdrFmAx9XrJSanDpNCAYY/zAT4C3gJ3Ai8aY7SJyv4hcZSebA+wWkT1AJrDIXv8S8DmwFaufYbMx5nV724+AJ4B9dpr+7VBurAdfdYc1hOaxB5eNuoyNB2o4d2SyPghHKTXohDUOwRizAuvW0NB1vwh5/xLWyb/tfgGgw1t2jDH5wFldyWyfqiuxfnZQQ2gee/CVnMt5tOQA8/N0/IFSavDRqSua1R4/ICzft5wzks+gutrq1D5XO5SVUoOQBoRmtcXWzzZNRl9UfsH28u3MP30+Gw5U4HQIecOH9EMGlVKqb2lAaNYSEFrXED4s/BCAi0deTP7+Cs7MSiQ2SqeAUkoNPhoQmtWWAAKxaa1Wrylaw2lJp5EancHmgkodf6CUGrQ0IDSrLYa4NHAeu/r3+r1sKN7AjKwZbC+qxtsU1ICglBq0NCA062AMwsaSjfgCPmZmzWTDgQpAB6QppQYvDQjNOhilvLZoLW6Hm8mZk9lw4CjZQ2IYmhTdTxlUSqm+pQGhWQc1hDVFazg341xiXDHk769gSq42FymlBi8NCADGtKshlNaXsqdiDzOyZlBQ0UBJjU/7D5RSg5oGBABvJQQaW9UQ1h1eB9Cq/0ADglJqMNOAAB2OUl5TtIaU6BTOSDmD/ANHife4GDc0sZ8yqJRSfU8DArQbpRw0QdYWrWX6sOk4xEH+/grOGTEEp05op5QaxDQgQLsawp6KPZR7y5mZNZNqbxO7i2s4d4Q2FymlBjcNCBASEKwawpqiNQDMyJrB5kOVGKP9B0qpwU8DAlhNRs4oiLYmrVtTtIbTh5xORmwGWwqqADg7Rye0U0oNbhoQ4NgYBBEa/A1sLN7IrKxZAGwtqGJkaixJse5+zqRSSvUtDQjQagzChuINNAWbmJk1E4CthVVMzE7qz9wppdRJoQEBWo1SXlO0hihHFOdmnktZrY/CygYm5WhAUEoNfhoQoFUNYW3RWiZnTibaFc3WQqv/YGK29h8opQY/DQjBANSXQXwmxXXF7Kvcd6y5yO5QPitbB6QppQY/DQh1ZWCCEJ/B+uL1AEzPmg7AloIqRqfHkRCtHcpKqcFPA0LIozOLaosAGJU0CoCthZV6u6lSKmJoQAgZpVxaX0pCVAIep4fiai/F1T69w0gpFTE0IITMY1TaUEpGjNW53Nx/oHcYKaUihQaE5oAQZwWEtNg0ALYUVuEQmJClHcpKqcigAaG2BKISICqWsvoy0mPSAdhaUMmYjARio1z9nEGllDo5NCDYYxCMMZQ2lJIem44xxhqhrM1FSqkIElZAEJF5IrJbRPaJyN0dbB8pIitFZIuIvC8iOfb6i0RkU8jLKyJX29ueFpEvQ7bl9W7RwmSPUq7yVdEUbCI9Jp3DVV7Kahu1/0ApFVE6DQgi4gT+DFwKTABuFJEJbZItBp41xkwC7gceBDDGrDLG5Blj8oC5QD3wdsh+dzZvN8Zs6nlxusGuIZQ2lAKQHpPeMsOp3mGklIok4dQQpgL7jDFfGGMageeB+W3STABW2u9XdbAd4HrgDWNMfXcz2yfsGkJLQIhNZ2thJS6HMH6YdigrpSJHOAEhGzgUslxgrwu1GbjOfn8NkCAiqW3SfANY2mbdIruZ6WER8XT04SJym4jki0h+aWlpGNntgqYG8FVZNYT61jWEsZkJRLudvft5Sil1CgsnIHT0IGHTZnkBMFtEPgNmA4WAv+UAIsOAicBbIfssBMYB5wEpwF0dfbgx5jFjzBRjzJT09PQwstsFoYPS7BpCanQqWwurtP9AKRVxwgkIBcDwkOUcoCg0gTGmyBhzrTHmHOBee11VSJIbgFeMMU0h+xw2Fh/wFFbT1MkVEhDKGsqId8dTXiNU1jfpHUZKqYgTTkBYD4wRkVEiEoXV9PNaaAIRSROR5mMtBJ5sc4wbadNcZNcaEBEBrga2dT37PRQySrmkvoS0mDS2FFYCMEmnvFZKRZhOA4Ixxg/8BKu5ZyfwojFmu4jcLyJX2cnmALtFZA+QCSxq3l9EcrFqGKvbHPo5EdkKbAXSgF/1qCSUpJ2LAAAZe0lEQVTdETKxXVlDmdWhXFBFlNPB2KHxJz07SinVn8IahmuMWQGsaLPuFyHvXwJeOs6++2nfCY0xZm5XMtonaksAgbg0SutLmZg+kS27qhg3LAGPSzuUlVKRJbJHKtcWQ2wqxuGyRinHpLNNn6GslIpQER4QrDEINU01+AI+nMEkanx+vcNIKRWRIjwgWKOUy+rLAKhriAX0GcpKqcgU2QGhrvUo5Saf1ZE8PCWmP3OllFL9InIDgjF2k5F1yymAvzEBt1OI9+iU10qpyBO5Zz5fNfi9LbecAjR4Y0mJM1hDI5RSKrJEbg2hzbQVMa4YqutdJMdG9W++lFKqn0RwQAh5lnK9dctpRV0jqfEaEJRSkUkDgl1DSItJo6K+SWsISqmIFcEBobnJKIOyhjIyYjMor/WRGqcBQSkVmSI4IBSDww0xyZTUl5ASnUa110+yBgSlVISK3IBQUwzxmdT562nwNxDnSgbQGoJSKmJFbkCoPQIJmS1PSosRKyBoDUEpFakiNyDUFEP80JZRyk5jzV+UogFBKRWhIjcgtKkh4E8ENCAopSJXZAYEfyPUl7eqIQT8CYAGBKVU5IrMgFBn33KaYE1b4XF6qKt3A+g4BKVUxIrMgFDTPChtaMuzlCsbmkiMduF2RuZXopRSkXn2qz1i/bRrCOkx6ZTXNWpzkVIqokVmQKixA4Ldh5Aea81jpAFBKRXJIjMg1BYDAnHpLRPbaQ1BKRXpIjMg1ByBuHQaTBO1TbVaQ1BKKSI1INRa01Y0P0s5LSaNo3WNOkpZKRXRIjMg1FiD0koarNtPE1ypNAaCOo+RUiqiRWZAqG09bYXbnrZCxyAopSJZ5AWEYMB6FkLCsSYjsQOCPi1NKRXJIi8g1JeDCbTUENwON40+D6A1BKVUZAsrIIjIPBHZLSL7ROTuDraPFJGVIrJFRN4XkRx7/UUisink5RWRq+1to0TkExHZKyIviMjJORvXHBuUVlrf/OhMPwCpcZ6TkgWllDoVdRoQRMQJ/Bm4FJgA3CgiE9okWww8a4yZBNwPPAhgjFlljMkzxuQBc4F64G17n4eAh40xY4AK4Hu9UJ7O1R6btqJ5UNrROh8AyXHuk5IFpZQ6FYVTQ5gK7DPGfGGMaQSeB+a3STMBWGm/X9XBdoDrgTeMMfUiIlgB4iV72zPA1V3NfLfUtJ+24mhdE26nEO9xnZQsKKXUqSicgJANHApZLrDXhdoMXGe/vwZIEJHUNmm+ASy136cClcYY/wmOCYCI3CYi+SKSX1paGkZ2O1F7bNqK5ontjtb5SImLwopTSikVmcIJCB2dJU2b5QXAbBH5DJgNFALNJ3tEZBgwEXirC8e0VhrzmDFmijFmSnp6ehjZ7URNMUQn4XMI1Y3VZMRmcLSuSTuUlVIRL5w2kgJgeMhyDlAUmsAYUwRcCyAi8cB1xpiqkCQ3AK8YY5rs5TJgiIi47FpCu2P2mdojED+UsgbrltP0mHQq6hv1llOlVMQLp4awHhhj3xUUhdX081poAhFJE5HmYy0EnmxzjBs51lyEMcZg9TVcb6/6DrC869nvhpriVo/ObJm2QmsISqkI12lAsK/gf4LV3LMTeNEYs11E7heRq+xkc4DdIrIHyAQWNe8vIrlYNYzVbQ59F/AfIrIPq09hSY9KEi67htA8StlqMmrUaSuUUhEvrNtqjDErgBVt1v0i5P1LHLtjqO2+++mgw9gY8wXWHUwnjzHtaghDPClUNezTie2UUhEvskYqeysh4GvpQ3CKEwLxAFpDUEpFvMgKCM3PUk4YSm1TLXHuOCrtUcpaQ1BKRbrICggtYxAy8QV8RDujOVrXCKAPx1FKRbzICgghNQSv34vH5dGAoJRStsgKCG1rCK5ojtZrQFBKKYi0gFBTDO5Y8CTgDXitJqNaKyDoOASlVKSLrIBQewTiM0EEn9+Hx+mhor6RxGgXbmdkfRVKKdVWZJ0Fa4qtgAD4Aj48Lg/ldY3aXKSUUkRaQKg9AglWQGhuMqrQgKCUUkCkBYSaYogfCmDdZeTUGoJSSjWLnIDQWAeNNS01BJ/fustIawhKKWWJnIBQc+zBOGA1GXkc1jgEHaWslFKRFBCan6WccKxT2SFRNAaCOo+RUkoRiQEhfijGGHwBHyZoTfaqYxCUUirM6a8HhZBpK3wBHwB+v1V8fVqaUidfU1MTBQUFeL3e/s7KoBEdHU1OTg5ut7tb+0dOQKg9Ag4XxKTga6oBwB9wAlpDUKo/FBQUkJCQQG5uLiIdPWZddYUxhvLycgoKChg1alS3jhE5TUbNg9IcDrx+64qksckKCKlxnv7MmVIRyev1kpqaqsGgl4gIqampPapxRU5AaJ62AlqajHyNdg0hrnvVK6VUz2gw6F09/T4jJyDUFEPCsVtOAbyNDqKcDuI9kdNyppRSxxM5ASG0huC3aggNPgfJcW69SlFKAXDrrbeyY8eOPv2Myy67jMrKynbr77vvPhYvXtynn92ZyLg09jdCfXm7GkKtT0jR/gOllO2JJ57o889YsWJFn39Gd0VGQKgrsX626UOo8wop2n+gVL/75evb2VFU3avHnJCVyH9deeZxt9fV1XHDDTdQUFBAIBDg5z//OY8++iiLFy9mypQpLFmyhIceeoisrCzGjBmDx+PhkUce4ZZbbiEmJoZdu3Zx4MABnnrqKZ555hnWrl3LtGnTePrppwFYunQpv/71rzHGcPnll/PQQw8BkJubS35+PmlpaSxatIhnn32W4cOHk56ezuTJk3v1O+iqyGgyChmDAMeajGrqtYagVKR68803ycrKYvPmzWzbto158+a1bCsqKuKBBx5g3bp1vPPOO+zatavVvhUVFbz33ns8/PDDXHnlldxxxx1s376drVu3smnTJoqKirjrrrt477332LRpE+vXr+fVV19tdYwNGzbw/PPP89lnn/Hyyy+zfv36k1LuE4mMGkLIozPhWJNRVT2kxGoNQan+dqIr+b4yceJEFixYwF133cUVV1zBBRdc0LLt008/Zfbs2aSkpADwta99jT179rRsv/LKKxERJk6cSGZmJhMnTgTgzDPPZP/+/Rw4cIA5c+aQnp4OwLe+9S0++OADrr766pZjfPjhh1xzzTXExsYCcNVVV/V5mTsTGQGheWK75hqC3WRU0yA6sZ1SEWrs2LFs2LCBFStWsHDhQi655JKWbcaYE+7r8VgtCw6Ho+V987Lf78flCu/Ueqrd0BIZTUa1xYBAXAZAy8A0jEsntlMqQhUVFREbG8tNN93EggUL2LhxY8u2qVOnsnr1aioqKvD7/SxbtqxLx542bRqrV6+mrKyMQCDA0qVLmT17dqs0F154Ia+88goNDQ3U1NTw+uuv90q5eiJyaghxaeC0ittcQzBBt9YQlIpQW7du5c4778ThcOB2u3n00UdZsGABANnZ2dxzzz1MmzaNrKwsJkyYQFJSUtjHHjZsGA8++CAXXXQRxhguu+wy5s+f3yrNueeey9e//nXy8vIYOXJkqyarfmOM6fQFzAN2A/uAuzvYPhJYCWwB3gdyQraNAN4GdgI7gFx7/dPAl8Am+5XXWT4mT55suuW5G4z5y6yWxb9s+os56+mzzMi7XjMf7yvt3jGVUj2yY8eO/s7CCdXU1BhjjGlqajJXXHGFefnll/s5R+Hp6HsF8k0Y5/pOawgi4gT+DFwMFADrReQ1Y0zo6I3FwLPGmGdEZC7wIHCzve1ZYJEx5h0RiQeCIfvdaYx5Kfzw1U3n3QqNtS2LPr8Pp7gAhz4tTSnVofvuu493330Xr9fLJZdc0qpDeLAKp8loKrDPGPMFgIg8D8zHutpvNgG4w36/CnjVTjsBcBlj3gEwxtTSH8Zc3GrRF/DhFCsQaEBQSnWkv0cN94dwOpWzgUMhywX2ulCbgevs99cACSKSCowFKkXkZRH5TER+Z9c4mi0SkS0i8rCInLQBAd6AFydWINCpr5VSyhJOQOjovqi292QtAGaLyGfAbKAQ8GPVQC6wt58HjAZusfdZCIyz16cAd3X44SK3iUi+iOSXlpaGkd3O+fw+HLiJ97hwOyPjRiullOpMOGfDAmB4yHIOUBSawBhTZIy51hhzDnCvva7K3vczY8wXxhg/VlPSufb2w3Z/hw94Cqtpqh1jzGPGmCnGmCnNgzx6yhvwIkQR7dZgoJRSzcI5I64HxojIKBGJAr4BvBaaQETSRKT5WAuBJ0P2TRaR5jP5XOy+BxEZZv8U4GpgW08K0hW+gA+HcROltQOllGrR6RnRvrL/CfAW1q2jLxpjtovI/SLSPNZ6DrBbRPYAmcAie98AVnPRShHZitX89Li9z3P2uq1AGvCrXitVJ3x+H4Ibj9vZeWKl1KBUWVnJX/7yl27vP2fOHPLz83sxR/0vrIFpxpgVwIo2634R8v4loMPbR+07jCZ1sH5ul3Lai7wBLxg3HpfWEJSKVM0B4fbbb+/vrJwyImOkchu+gA+MRwOCUqeKN+6GI1t795hDJ8Klvznu5rvvvpvPP/+cvLw8LrroIrZs2UJFRQVNTU386le/Yv78+ezfv59LL72U888/nzVr1pCdnc3y5cuJiYkB4B//+Ae33347lZWVLFmy5NQYbdwDERkQvH4vJhiPx6VNRkpFqt/85jds27aNTZs24ff7qa+vJzExkbKyMqZPn94y++jevXtZunQpjz/+ODfccAPLli3jpptuAsDv9/Ppp5+yYsUKfvnLX/Luu+/2Z5F6LCIDglVDcOPRu4yUOjWc4Er+ZDDGcM899/DBBx/gcDgoLCykuNh6jsqoUaPIy8sDYPLkyezfv79lv2uvvbbD9QNVxAaEYNBFlAYEpRTw3HPPUVpayoYNG3C73eTm5uL1WrMih05v7XQ6aWhoaFlu3uZ0OvH7/Sc3030gIs+IXr+XYMClNQSlIlhCQgI1NTUAVFVVkZGRgdvtZtWqVRw4cKCfc9c/IraG4A64tQ9BqQiWmprKrFmzOOusszjvvPPYtWsXU6ZMIS8vj3HjxvV39vpFxAWEpmATARPAEXTqXUZKRbi///3vnabZtu3YmNnm5yUAvP/++y3v09LSBkUfQsSdEX1+6+E4fr9LA4JSSoWIuDOiN2B1FAUCLqI0ICilVIuIOyM2Pz6zye/UPgSllAoReQHBbjIKBnXqCqWUChVxZ8TmJiOCetupUkqFirgzYnOTkTF626lSSoWKuIDg9TfXELTJSCnV2q233sqOHTs6T9iL5s2bx5AhQ7jiiivabVu6dCmLFi3iueeeY9KkSUyaNImZM2eyefPmPslLxI1DCK0h6F1GSqlQTzzxxEn/zDvvvJP6+nr++te/ttv25ptv8q//+q/4fD5Wr15NcnIyb7zxBrfddhuffPJJr+cl4gLCsT4EbTJS6lTx0KcPsevorl495riUcdw1tcNHtQNQV1fHDTfcQEFBAYFAgJ///Oc8+uijLF68mClTprBkyRIeeughsrKyGDNmDB6Ph0ceeYRbbrmFmJgYdu3axYEDB3jqqad45plnWLt2LdOmTePpp58G4Ec/+hHr16+noaGB66+/nl/+8pcd5uMrX/lKq0FuzYwxbNq0iXPPPRfrwZKW6dOnU1BQ0KPv5ngi7hK5+S4jY3RgmlKR7M033yQrK4vNmzezbds25s2b17KtqKiIBx54gHXr1vHOO++wa1frYFVRUcF7773Hww8/zJVXXskdd9zB9u3b2bp1K5s2bQJg0aJF5Ofns2XLFlavXs2WLVu6lL/PPvuMs88+u1UwAFiyZAmXXnppN0t9YhFXQ2huMtLpr5U6dZzoSr6vTJw4kQULFnDXXXdxxRVXtHq4zaeffsrs2bNJSUkB4Gtf+xp79uxp2X7llVciIkycOJHMzEwmTpwIwJlnnsn+/fvJy8vjxRdf5LHHHsPv93P48GF27NjBpEntHh55XG+++Wa7E/+qVatYsmQJH330UU+KflwRd0Zs7lQ22mSkVEQbO3YsGzZsYOLEiSxcuJD777+/ZZsx5oT7Nk977XA4Wk2P7XA48Pv9fPnllyxevJiVK1eyZcsWLr/8crxeL5988gl5eXnk5eXx2muvnfAz3n77bS655JKW5S1btnDrrbeyfPlyUlNTu1PkTkV0DUE7lZWKXEVFRaSkpHDTTTcRHx/f0vYPMHXqVO644w4qKipISEhg2bJlLbWAcFRXVxMXF0dSUhLFxcW88cYbzJkzh2nTprU0KZ1IVVUVfr+/5cR/8OBBrr32Wv72t78xduzYLpc1XBEXELwBL4KA0dlOlYpkW7du5c4778ThcOB2u3n00UdbZjPNzs7mnnvuYdq0aWRlZTFhwgSSkpLCPvbZZ5/NOeecw5lnnsno0aOZNWvWcdNecMEF7Nq1i9raWnJycliyZAk1NTV89atfbUlz//33U15ezu233w6Ay+UiPz+/myU/PumsanQqmTJliunpl7B4/WL+vut5jm7/Je/952xGp8f3Uu6UUl2xc+dOxo8f39/ZOK7a2lri4+Px+/1cc801fPe73+Waa645KZ996623cuuttzJ9+vQu79vR9yoiG4wxUzrbNyJrCC6x2vw8bu1DUEp17L777uPdd9/F6/VyySWXcPXVV5+0z+6P8RAQgQHBF/DhkigAbTJSSh3X4sWL+zsLJ13EnRF9fh9ODQhKnRIGUpP1QNDT7zPizojegBcnVkDQu4yU6j/R0dGUl5drUOglxhjKy8uJjo7u9jEissnIgRuAKKcGBKX6S05ODgUFBZSWlvZ3VgaN6OhocnJyur1/xAUEr9+LEIXH5Wg3JFwpdfK43W5GjRrV39lQIcK6RBaReSKyW0T2icjdHWwfKSIrRWSLiLwvIjkh20aIyNsislNEdohIrr1+lIh8IiJ7ReQFEbthv4/5Aj7E6NTXSinVVqdnRRFxAn8GLgUmADeKyIQ2yRYDzxpjJgH3Aw+GbHsW+J0xZjwwFSix1z8EPGyMGQNUAN/rSUHC5Qv4ENx6y6lSSrURzmXyVGCfMeYLY0wj8Dwwv02aCcBK+/2q5u124HAZY94BMMbUGmPqxWqrmQu8ZO/zDHBSbvL1+r3WtBXaf6CUUq2E04eQDRwKWS4AprVJsxm4DvgjcA2QICKpwFigUkReBkYB7wJ3A8lApTHGH3LM7I4+XERuA26zF2tFZHcYee5IGlB2bPExZGE3j3RqaFOeAW0wlQUGV3kGU1kgcsszMpyDhRMQOup5bXuf2ALgERG5BfgAKAT89vEvAM4BDgIvALcAHU3z1+G9Z8aYx4DHwsjnCYlIfjhDtweKwVSewVQWGFzlGUxlAS1PZ8JpNykAhocs5wBFoQmMMUXGmGuNMecA99rrqux9P7Obm/zAq8C5WBFtiIi4jndMpZRSJ1c4AWE9MMa+KygK+AZtrvBFJE1Emo+1EHgyZN9kEUm3l+cCO4w1EmUVcL29/jvA8u4XQymlVE91GhDsK/ufAG8BO4EXjTHbReR+EbnKTjYH2C0ie4BMYJG9bwCrOWmliGzFan563N7nLuA/RGQfkAos6bVSdazHzU6nmMFUnsFUFhhc5RlMZQEtzwkNqOmvlVJK9R2991IppRSgAUEppZQtIgJCZ1NvnMpE5EkRKRGRbSHrUkTkHXvaj3dEJLk/89gVIjJcRFbZU5lsF5F/s9cPuDKJSLSIfCoim+2y/NJe3y/TsvQWEXGKyGci8k97eUCWR0T2i8hWEdkkIvn2ugH3d9ZMRIaIyEsissv+/5nR2+UZ9AEhzKk3TmVPA/ParLsbWGlP+7HSXh4o/MB/2lOZTAd+bP8+BmKZfMBcY8zZQB4wT0Sm00/TsvSif8O6gaTZQC7PRcaYvJB79Qfi31mzPwJvGmPGAWdj/Y56tzzGmEH9AmYAb4UsLwQW9ne+uliGXGBbyPJuYJj9fhiwu7/z2IOyLQcuHuhlAmKBjVij+Muwpmxp9/d3qr+wxgStxLpF/J9YdwYOyPIA+4G0NusG5N8ZkAh8iX0jUF+VZ9DXEOh46o0Op8kYQDKNMYcB7J8Z/ZyfbrFnvj0H+IQBWia7eWUT1qSN7wCfE+a0LKeo/wZ+BgTt5VQGbnkM8LaIbLCnwIEB+ncGjAZKgafs5rwnRCSOXi5PJASEcKbeUCeZiMQDy4B/N8ZU93d+ussYEzDG5GFdWU8FxneU7OTmqntE5AqgxBizIXR1B0kHRHmAWcaYc7Gai38sIhf2d4Z6wIU1y8OjxpoRoo4+aO6KhIDQ6dQbA1CxiAwDsH+WdJL+lCIibqxg8Jwx5mV79YAukzGmEngfq19koE7LMgu4SkT2Y81qPBerxjAgy2OMKbJ/lgCvYAXsgfp3VgAUGGM+sZdfwgoQvVqeSAgInU69MQC9hjXdBwywaT/sqc+XADuNMX8I2TTgyiQi6SIyxH4fA3wVq6NvQE7LYoxZaIzJMcbkYv2fvGeM+RYDsDwiEiciCc3vgUuAbQzAvzMAY8wR4JCInGGv+gqwg94uT393lpykDpnLgD1Y7bv39nd+upj3pcBhoAnrKuF7WO26K4G99s+U/s5nF8pzPlaTwxZgk/26bCCWCZgEfGaXZRvwC3v9aOBTYB/wD8DT33ntRtnmAP8cqOWx87zZfm1v/r8fiH9nIWXKA/Ltv7dXsR4j0Kvl0akrlFJKAZHRZKSUUioMGhCUUkoBGhCUUkrZNCAopZQCNCAopZSyaUBQSikFaEBQSill+//278kjvB+xtgAAAABJRU5ErkJggg==\n",
      "text/plain": [
       "<Figure size 432x288 with 1 Axes>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "accuracies_shifted_sig = [\n",
    "    0.9489, 0.9691, 0.9763, 0.9789, 0.9808, 0.9823, 0.9841, 0.9845, 0.9846, 0.9854, 0.986, 0.9863, 0.987, 0.9873,\n",
    "    0.9873, 0.9872, 0.9875, 0.9876, 0.9881, 0.9879, 0.9877, 0.9876, 0.9876, 0.9875, 0.9877, 0.9876, 0.9875, 0.9873,\n",
    "    0.9875, 0.9875, 0.9875, 0.9875, 0.9873, 0.9874, 0.9873, 0.9875, 0.9876, 0.9875, 0.9875, 0.9876, 0.9876, 0.9876,\n",
    "    0.9876, 0.9876, 0.9876, 0.9876, 0.9876, 0.9876, 0.9876, 0.9876, 0.9876, 0.9877, 0.9877, 0.9876, 0.9876, 0.9876,\n",
    "    0.9875, 0.9875, 0.9876, 0.9876\n",
    "]\n",
    "\n",
    "plt.ylim(bottom=0.96, top=0.99)\n",
    "plt.plot(accuracies_sigmoid, label=\"sigmoid\")\n",
    "plt.plot(accuracies_tanh, label=\"tanh\")\n",
    "plt.plot(accuracies_shifted_sig, label=\"sigma-1/2\")\n",
    "plt.legend()\n",
    "plt.show()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "We learn faster than the standard sigmoid but a little slower than $\\tanh$, but we reach better stabilized accuracies! (Again, I've only trained one network, so those results shouldn't be over-interpreted).\n",
    "\n",
    "\n",
    "What about multiplying the derivative by $2$, to get closer to the derivative of $\\tanh$? Let's try $2\\sigma(z) - 1$ (`exec_shifted_2sig.py`):"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 5,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "image/png": "iVBORw0KGgoAAAANSUhEUgAAAYQAAAD8CAYAAAB3u9PLAAAABHNCSVQICAgIfAhkiAAAAAlwSFlzAAALEgAACxIB0t1+/AAAADl0RVh0U29mdHdhcmUAbWF0cGxvdGxpYiB2ZXJzaW9uIDIuMi4yLCBodHRwOi8vbWF0cGxvdGxpYi5vcmcvhp/UCwAAIABJREFUeJzs3Xl4VOXZ+PHvPUsm2yRkh6yAgiyCESKIoCgW61KqqMVqtbXV2n2x1Sr681VRtLa01r629rXurWvdte64IIvKvq9qdshG9mQmszy/P2ayR5iEhEDm/lxXLmbOec6Z50zIc59nPWKMQSmllLIMdgaUUkodGTQgKKWUAjQgKKWUCtKAoJRSCtCAoJRSKkgDglJKKSDEgCAiZ4vIThHZIyI39rA/R0SWisgmEflQRDI77LtHRLYEfy7psH2UiHwqIrtF5FkRieifS1JKKdUXBw0IImIF/gacA0wALhWRCV2SLQGeMMZMBhYBdwePPQ+YAuQC04HrRSQueMw9wL3GmDFANXDVoV+OUkqpvgqlhjAN2GOM+cIY0wI8A5zfJc0EYGnw9Qcd9k8APjLGeI0xjcBG4GwREWAO8Hww3ePABX2/DKWUUofKFkKaDKCow/tiAnf7HW0ELgLuA+YDThFJCm6/VUT+DEQDZwDbgCSgxhjj7XDOjJ4+XESuAa4BiImJmTpu3LgQsqyUUqrV2rVrK40xKQdLF0pAkB62dV3v4jrgfhG5ElgGlABeY8w7InISsBKoAFYB3hDPGdhozIPAgwB5eXlmzZo1IWRZKaVUKxEpCCVdKE1GxUBWh/eZQGnHBMaYUmPMhcaYE4Gbg9tqg/8uNsbkGmPmEggEu4FKYJiI2L7qnEoppQ6vUALCamBMcFRQBPBt4NWOCUQkWURaz7UQeCS43RpsOkJEJgOTgXdMYEW9D4CLg8d8D3jlUC9GKaVU3x00IATb+X8OvA1sB54zxmwVkUUi8s1gstOBnSKyC0gDFge324GPRWQbgWafyzv0G9wA/EZE9hDoU3i4n65JKaVUH8jRtPy19iEopVTvichaY0zewdLpTGWllFKABgSllFJBGhCUUkoBGhCUUkoFaUBQSikFaEBQSikVpAFBKaUUENpaRkr1isfv4dkdz9LkbWJmxkzGJ47HInrvodSRTgOC6lf5tfks/HghW6q2APC/6/+XxMhETkk/hZkZMzkl/RQSIxMHOZdKqZ5oQFD9whjD87uf54+r/0iENYI/zf4TU9OmsrJ0JStKV7CiZAWvf/E6gjA+aTwz02cyK2MWk1MmY7Pof0OljgS6dIU6ZFXNVdy28jY+LP6QGSNmcMfMO0iLSeuUxm/8bKvaxoqSFawoXcHGio34jR+n3cmJaScyMm4k2c5ssuKyyHZmMyJmBFaLtVf5qGiqYGXpSlbtXYUgnc6X7cwm3hFP4NlMKhwZY6hx11BYX0hhXSFF9UUU1hdS46rpltZutZMZm0l2XOD/TpYzixGxI7Bb7IOQ80MX6tIVGhBUyHx+H+vK1/Fl7ZcU1hVSWB/8o6orBODaqddy2fjLQuovqHXX8uneT1lRuoJNFZsoqi/C7XMf8BiLWBgRM4IsZ7CQj8smLSaNHVU7WFG6gh37dwCQGJmIw+pgX+M+TIfHbDjtTrLisshx5rQFimMTjj1oH0ejp5FtVdvw+D2hfE09So9JJycup18CktvnZmP5RvLr8jv9Horri3FGOAPfT2tBFpdFXERct3PUueu6FYyVzZWHnLejiSCMiBlBUlQS0uURLU3eJkoaSmj2Nvfp3MlRyW2BpPV3Eefo4ffQUkdRXVGn30VFc0WP53zlglcYHT+6T/nRgKD6TY2rhpf2vMSzO5+lpKEEAIfV0algPv+Y8zk24dg+f4bf+Kloqmgr3Moay/Dj75TG6/dS2lDaVoDVumsBsImN3NRcZmYEmqHGJozFIhbcPjcl9SUU1RdRUFdAUX1R27GlDaX4jA+ABEcCM9JnMCtjFjPSZ5AUmcSu6l1tTV3rytfh9Xu75bm3MmIzmJUxi1PST2H6iOnE2GNw+9wU1xe3Fex1LXVkxma2FSQpUYGHXOXX5bfVrtbsW4PL5wLafw9ZziwyYjNo9DQGvsO6Isqbyw+ap9So1LbgmBKdMuQ7/+Mi4toCZWZsJhHWiK9Ma4yhylXV9rvZ27gXv/F/ZfpWnf4v9+Pv4dJxl/a5/00Dgupmc8Vmntn5DNv3b++2z2FxkOnM7HRHYxELL+5+kTe+fAO3z01eWh6XHHcJuam5pEanDnrhUeuupbShlCxnFrERsb061uP3sLdhL5srN7cVtPtd+wGId8S3BZuxCWOZmTGTacOnEWOP6VM+jTHsrt7N8tLlfLr3U5q9zdgsNhIjE6loquhUixGk0/tIaySxEbFtd+8j40a2dc6PTRh7wN9Dk6eJovoimrxN3fbF2GPIjM0k2h7dp2tSoWvyNFHcUEyjp7HbvmhbNFnOrLbfg8fnp7qppcfzJEZHYLP27W9OA4ICAs0Lb+e/zdPbn2ZL1RaibdFMGz6tW/t863/ajnfOAFG2KOaNnse3x32bMQljDnf2Dxu/8bNj/w5WlKygoK6AqWlTmZkxk9To1H79HI/Pw/ry9SwvXU5VcxWZzsy2Po7suGxi7DHsa9zXdndZWF/Iftd+clNyOSXjFLKcWQf/EHXEaGrxUlDVREFVIw1uX7f9tc0eCqoayQ+mKa5uxufvuUx+7zezOTa1dzc+rTQghDljDP/a9i8e2vwQ1e5qRsWP4tJxlzJv9LwD3k233jkX1hdS565jVuasHtuglToaeXx+iqubya9qpKAyUBB/1R15X/kNlNW6yK9qpLz+wP1iAM5IGyOTYshJimZkUgxpcY4e+5rmTU4nPrpvndqhBgQd7zcEGWO4b919PLzlYWamz+TK469k+vDpIXVo2i32QJNRXPZhyOkA8LqhphBa2qvnXr+hot5NTFwCccOPAesA/bf3uAKf7eneRNOjiFgYlg22ntuxjTHUNHkorW0m1Pu2FKeDVGfPBcrh5Pb6KK5uDtz9Vgbufgsr62muLEB6GNXjsFkYHh9J+rAoRsRHkR4fSYrTgcXS+TosIqTEOnBGfvXv0O31U17vorTGRWlNM3trXZTWNrOv1kV5vbvTHXiU3cKw6Aj6+9saFRPBKVmRjBgWxYj4SEbER/WY56gIK3GRtg6d2s3Bnx7Y0wg8hHLgaEAYYowx/Hntn3ls62MsGLuAm0++edDb+g9Ffe1+yvK3UVuyE09VAaZL564YP7HucuKbi4hvLsLp3ofQufS0ASOCr31Y8cZl4Ug9FhJHQ2wa9LXw9Pugthj2fwH7v4S6EqB3NW4/Fhoih1MbmUVtVCbVthSqm33UNLVQ3dSC23PwTsyubFYhITqCYdF2EqIjcNgH/vdvDNS7vMF8e6h3eTDGkCK1jJJ9zLGUkynl2OjebAKAB6gM/hwiB5AV/Ommp/LUdeif2eM5q/r5nD9bDSlj+/mknWlAGEKMMfxh9R/49/Z/c+m4S1k4bWHod4p+P1hCKDg8LqgpCBaCwZ/q/MCd+SHyG9NWqLhdjSR79pJIHc6DHFdlnBSaNNaY0RSYGRT40/DYnaTGBe6WU50OUmMdVFbuozx/O+nVexnX+CU5+Z9g9zYcUp5rLfGUWdPZZzuOfc4zKLOOwGWJ6pTGGGhweSlvcOP1tQeMOBrJsZST07iPkU1lZMtWjpcu+enrDaEr+LO/j8cfqmDJ4rPHYhJGYU2ejiSODgTh6EQ4wD15k8dLRb2b6sbuw3y9fkNVg5vyejcV9W7K6l3sb2wh1mEj1ekgxRlJqjMiUFOKiyTNGUmMw9ptWOlRKW7EwdMcIg0IQ4Qxhrs/u5undzzN5eMv53cn/a5zMDAGmqra72b3f965UHfVBZovWv9oE0dDbGrgrrct3ZeBO+KOd8GOeEgcGWj+6CWvz4/L66fR7aWm2UNdswe/MQjgiIxmT+Js/MNG4Ug7lviMcaRkjSHCEdn5JCLEWiOYAEzosDnCaukxGDa4vby0rpgfrypgT3k9MVYfmQnR5CRFk50YTU5SDMOi7BTXNFNY1UjB/iYKq5q6tQUPi7KTlRiNNSKy22f0JD7Bzqjk9nbinKRokmMd3SonPr8Hq6X/Ci+/3+Dx976W0VuCEGHrfkNhtUb0ugYWDeQEf0Lh95tuTUuqb7RTeQjY17iP/13/v7z6+at8b8zF/Db9a0j1l50L/P1fgruu/SCxQHxme+EfGQ/VHe78O6aNTmpPlzAKko5pfx+V8JV/8MYY9je2tI2g6PpvTVP7HWBmQhSzx6Ywe2wKpxybTKxjYO9VjDGs+ryKj3ZXUFjV1Janppb2Jo0Up4ORSdFkJwYL8uQYRiZFk5MY0+fOPaUGg44yGgq2vAgf/xl6mC1pMKy2Gp6J8PG+LTBy/ar6Jn5RVdleORZr4K6/YwHeWqgn5IDN0e28NU0tfLyrgtXb97B/XxEZ2ccwfeJoTh6dRHTEwQtpr8/Pu9vKePLTQjYW1VDvbm/zF4H0+Khud8pj0pyMTIoe9I5QYwwVDYGmisyEKGIGOCgpdbhoQDiauWrhjd/BpmcgbVKnjiS/MbzgreBJTxmfGxfxWLnInsICWyoZca1NPqMC/8ZngfXAd7I+v2FTcQ0f7argo10VbCyqwW8gPsrOccOdbC6updnjI8Jq4aRRCZw2JoVxI+IYmRRNxrCotokylQ1unvmskCc/LWRvrYuMYVGcOT6VkUkxjEwONMVkJkThsPVufSKl1KHTYadHq4KV8OKPAm33s2+E065vGybZ5Gni5uU3817hGsYnjueO8Zdx9sizibSF1o7dqrzOxUe7Kli2u5KPd1dQ0+RBBE7IHMYv5oxh9nEpnJA5DKtFcHt9rMmvDgSMnRXc/eaOtvPYLEJmQhSpcZFsKKyhxefn1DHJLDr/eOaMS+3XtnClBoppaaGlpARPYSEtBYW0FBbiragg5LG+h0nazTdhT0s7eMJDoAHhSOHzwAd3wfJ7IWEk/OBtyDqpbXd5Uzm/eP8XbK/azvV513PFhCt63cTS3OLjdy9s4rWNpQAkxzo4c1was49LYdaxySTGdB8P77BZmXlsMjOPTeamc8dT2eDmy8pG8isbyQ/2BxRXN3PptCyumDGyzzMp1dDib2qipagIf1OIczIOBwO+mupgoV+AJ1j4e0pLA6PsgiwxMdjS0pA+LhMxUExL/06g64kGhCOBMfDqL2HjUzDlu/D1u8HRXrBuq9rGL97/BQ0tDfx1zl85Pev0Xn9EWZ2LHz6xhs0ltfx49jHMO2EE44fH9Xp0RnKsg+RYByeN/OpFtloKCmhctQpLVBQxM2diS07uMZ2vtpbGVavwlJRgz8gkIicbe1Y21ti+rRn0VTz79tG4fDmevfv69bwDQRwOIrIysWdnE5GdjdV5sEG33fldLjxFRbQE73j9jd3X0Olvxu/Du68s8JmFBfgqjuyVU63x8dhzcojKzSX+m/OIyMnBnpVNRE421sTEQe/PGiwaEI4Eqx8KBIPZN8IZCzvtWlq4lIUfL2SYYxhPnPMExyUe1+vTbymp5erH11Dn8vDgFXnMndC/1U5/YyONq1fTuOxjGpYvx1NY2Gl/5IQJxJx6KrGnzkIcDhqXL6dh2cc0b9zY6c6slTU5mYhggRiRk91WONozMxH7V69O2Z4hH66tW2n4eDmNH3+Me/fu/rrUw86amIg9PR2xHfxP1Rg/3rJyvPsGJ/DZUlOJyM4m9rTTiMjKJiI7C4vzyFr2xBofFwi0w4YNdlaOSCF1KovI2cB9gBV4yBjz+y77c4BHgBQCU2EuN8YUB/f9ATgPsADvAr8yxhgR+ZDABNLWITRnGWMOuE7skOxULvwEHjsPjv0afPvpTpPDXvv8NW5efjOTkidx35z7SI7q+U77QN7Zuo9fPbOBhGg7D33vJCak9+0PtLUJoKWgoFNba0thYaAAMgaJiiJm2jRiTj2VmJmn4G9sonF5IEg0r98AvuCQThEiJ04k5tRZxJ56Ko5jjsFTWtrhnAV48gtoKSrCW1bWp/wCiN1OVN5UYmedSsyps3CMGXPE3/kFvufiwHcQ/J67NmkciC01tUMQzSEiOwtrfPwA51od6fptlJGIWIFdwFygGFgNXGqM2dYhzX+A140xj4vIHOD7xpgrROQU4I/AacGky4GFxpgPgwHhOmNMyCX8kAsI9fvg/06DiBj44QcQ1X7XsrJ0JT9772dMTZvK/Wfe3+uOY5fHx/999AV/WbqLyZnD+Od3p5LqDO0c3ooKGpavoGn1aloKCnpsArAmJRGRldVW+ESfeCJRU6dicXQfygrgq6+ncdUqTIuHmFNmYEsMbV13f3MzLUVFeAoL8ZSUYHyhFYwRo0YSM306lmhd3lmp/hxlNA3YY4z5InjiZ4DzgW0d0kwArg2+/gB4OfjaAJFABIG56nag77d8Q4m3BZ77Lrjr4YqXOwWD7VXbufaDaxk9bDT3nnEvkbZIGtxeVu6p5KNdFawrrGHayIQeO3GNMby+aS/3vLWD4upmzs9N556LJhNpDwz39LvdgTvOrtkprwg05Sxfjnt74HkJ1oQEHMccE2gCyM4JtvFnEZGTgzW2d53HVqeTuLPO6u23hCUqisixY4kcO7BruCilQgsIGUBRh/fFwPQuaTYCFxFoVpoPOEUkyRizSkQ+APYSCAj3G2M6Pp3lURHxAS8Ad5qjaVLEoXr7Jij6FC5+FNLaF10oaSjhp0t/Spwjjr/Mvp+nPynn/R1bWJNfjddviImwMjEjnqc/K+LxVQXMOjaZ756czclVu/mixs3iAjurSxsZN9zJk1dPZ+axgWamluJiqp9+mtrnX8BXW9tznmw2ok88kZTf/IbYU2fhOO44JJT1jZRSQ0IoAaGnRteuBfd1wP0iciWwDCgBvCJyLDAeyAyme1dETjPGLAO+Y4wpEREngYBwBfBEtw8XuQa4BiA7+yhdkrmrTc/B6n/CKb+A4y9s21zjquHH7/4Yt8/N/33tn9zxSgnvbCtj/Ig4rj51NLPHpjA1J4EIm4XKBjfPri7ilQ+3UPTvP5K1bysO4CZbBC2TpnDspLk4I0bSsHwF1U8+ScOHH4LFgvNrXyP2jNORLktAW5yxROfl9frOXyk1dIQSEIrpvJJsJtCpzcEYUwpcCCAiscBFxpjaYGH+iTGmIbjvTeBkYJkxpiR4bL2IPEWgaapbQDDGPAg8CIE+hN5d3hGoaT+8eQNkTYczbwMCT9EqbijmlhW3UNpQyv/N/T/+tczFO9vKuHXeBL4/c1S30yTHOviepYS57y3BW1vHR2deSuwxo5hd+znuFSsov+MOWnvorUlJJP34RyRccgn24cMP37UqpY4qoQSE1cAYERlF4M7/28BlHROISDKw3xjjBxYSGHEEUAj8UETuJlDTmA38RURswDBjTKWI2IFvAO/1xwUd8T5YzCqaWTpqIoVLf9r28O6kah8ztxt+kncpO/67l1e3NPKjr03uMRj4m5sp/+MfqX7qaRxjx5L9yMNMOK59OKoxBk9BAQ0rVmAdNgzn3LlYIkIYrqmUCmuhDjs9F/gLgWGnjxhjFovIImCNMeZVEbkYuJtAU9Iy4GfGGHdwhNLfCYwyMsBbxpjfiEhMMJ09eM73gN8YY77i6RkBR/soo893v8GS937J8ugoYu2xjIwbSVZcFuPqnZx01+vYquo6pbfExGDPyOg2Bt1bWYm3vJzEK68k5dpff+XIHqWUAl3c7ohS1VzFAxv+zvM7nyPaGH504s+5dNIPiLBG4P78cwquvBJ8fipvuYcb3/ic02Nd/Py4SPwlJYERQb7OcVLsNhIuu4yYGTMG54KUUkcVXdzuCPH6F6+z+JPFNHsaWVBXz09OvpGE3B8D4N69m4Irvw8i7Lv9Xn7y8X6yjhnFdT+eQVykrrevlDq8dEzhAHJ5Xdz16V2MdGbx4n43N0WOIiHvmsC+nTsp+O738CH8/Zu/4btLK0iLj+TxH0zTYKCUGhRaQxhA7xa8S31LPb82wxhdWwYLngKLBdf27eRf+QMajIVf511FXVM0t3xjLFecnNPjYwiVUupw0IAwgF7Y/QLZ0cM5acMLMOUKyJhK85atfP6977PfWLlp1o+YO3cqv5wzhoQelp5WSqnDSQPCAPmi9gvWlq3l1y4rlogYOPNWmjdt4svvX0WlsfPvBTfwxFVnMjpFJ4IppY4MGhAGQvl2XnzrR9iM4Xy3HxY8QdOuIgqvvoZKcfDnc3/F4784R2sFSqkjijZY96eGCnj9WloeOIVXvVWcHjua5J+upqnaSeFVV1Npj2bhaT/lrh+fpcFAKXXE0RpCf6krhQfPgKZK3p/0DarrN3DxjBto2rCZwmt+RF3MMH415Sr+3xWncXyGrk+vlDryaA2hP3jd7UtZ//B9XoiNJD0mnWlR4yn+1a9xD0viR1OvZt6ZJzD/xMyDn08ppQaBBoT+8NaNULwaLvg7RTGJfLL3E+aPmU/FPX/AV1PD7yZewphxI7n5vPGDnVOllPpKGhAO1bp/wZpHYOavYOIFvLjnRSxi4bzydGpfeYXXJs6lZkQO93/nROxW/bqVUkcu7UM4FCXr4L+/hdGnw5z/weP38PKelzkz6RQa7ryX0rg0np8wl8e/Py3kx1cqpdRg0VvWvmqshGevgNhUuOgRsNpYVryMyuZKzn/Pi7+sjIdmXMq/f3Iq40f07cH2Sil1OGkNoS/8fnj++9BYAVe9DTFJALyw6wVOKosj+c0VvDvuNO64+TKOTXUOcmaVUio0WkPoi6JP4MtlcNadkH4iAJ/XfM4nBcu47EU3+2MS+MZ9izQYKKWOKlpD6ItNz4E9GnLbHxz3xMq/c/lSIaOmkYg//Z5jclIHMYNKKdV7GhB6y9sC217GjD0X1/Y9NHy8nOqP3mfBlu1YgKa55zH+vK8Ndi6VUqrXNCD01p738NXVkP/AblpKLwUR9o9O4r1ZVj5P+iGPLv75YOdQKaX6RANCb216loaqZFpKK0i7aSHeuTP5znsLaNx/EtdPmYfVZh3sHCqlVJ9op3JvuOpg11vUV2djTU4m4Tvf4em9r+Pxe5DaM/jW1KzBzqFSSvWZBoTe2P4afpeLhp37cX7tTBp8TTy9/Rn89ZO4YGIu8dH66Eul1NFLA0JvbH6OxoZsjMuNc+5cnt35LI3eBporZnPFjJzBzp1SSh0SDQihqt8HXy6jbn8mlvh4LFMm8cTWJ7C7x3Pi8OOZmK5LWiuljm4aEEK15QWM10/Dtgqcc+bwcv7rVLurqd17Kt/V2oFSagjQgBCqTc/S6B2Pv6GR6Lln8tjWx4g1xzLMOo5zjh8x2LlTSqlDpgEhFBW7YO9G6qsysMTEsGkk7G3cS0XJyVw2LZsIm36NSqmjn5Zkodj8HMZYqN9UQuzs2bxc+F8cEodpnMBl07MHO3dKKdUvdGJaV83V4PN22GBg839osubhqynGcsZMPiy6E2/tDOaOT2dEfNSgZVUppfpTSAFBRM4G7gOswEPGmN932Z8DPAKkAPuBy40xxcF9fwDOI1AbeRf4lTHGiMhU4DEgCnijdXt/XFSf7XwTnv52j7vq909GIitZllWPZ7+Hxspczj89/TBnUCmlBs5BA4KIWIG/AXOBYmC1iLxqjNnWIdkS4AljzOMiMge4G7hCRE4BZgKTg+mWA7OBD4EHgGuATwgEhLOBN/vjovqsPHhJ5/wBpL01zVgjqb/2IWJPncVLxW+SHDGKenc6eSMTBymjSinV/0KpIUwD9hhjvgAQkWeA84GOAWECcG3w9QfAy8HXBogEIgAB7ECZiIwA4owxq4LnfAK4gMEOCA3l4IiH6T/qtLl5/Xq85eU0zpzM1qq/MpJLyUmKJsXpGKSMKqVU/wulUzkDKOrwvji4raONwEXB1/MBp4gkBQv8D4C9wZ+3jTHbg8cXH+ScAIjINSKyRkTWVFRUhJDdQ9BQBrEp3TbXv/Mu2O28lV6JTWyUFI9nak7CwOZFKaUOs1BqCNLDtq5t/dcB94vIlcAyoATwisixwHggM5juXRE5DWgO4ZyBjcY8CDwIkJeXN6B9DO78YsqXgb16EfbsbCKyc4jIyab+3XeJPnk6L+19m7zUmby7LUIDglJqyAklIBQDHZfxzARKOyYwxpQCFwKISCxwkTGmVkSuAT4xxjQE970JnAz8i/Yg0eM5B0Pt+jIa9nixlP0Xf11dp337LzmDKtcnfC11NgB5Odp/oJQaWkIJCKuBMSIyisCd/7eByzomEJFkYL8xxg8sJDDiCKAQ+KGI3E2gpjEb+IsxZq+I1IvIycCnwHeB/+2PCzoUzSVuIrOSGPXOKnw1NbQUFtJSUIivuppHRqwhcX8i9fuPIS6ygjGpsYOdXaWU6lcH7UMwxniBnwNvA9uB54wxW0VkkYh8M5jsdGCniOwC0oDFwe3PA58Dmwn0M2w0xrwW3PcT4CFgTzDNoHYom8ZamistRI8NVFysw4YRNXky8fO+gWXBPJbu+5hzR53LuoJ6puQkYLH01JKmlFJHr5DmIRhj3iAwNLTjtv/p8Pp5AoV/1+N8wI+6bg/uWwMc35vMDiTX+k8wPiHq+LHd9r2Z/yYev4czM8/jgfICzs/V+QdKqaFHl64IalrzKQBRuSd22/fKnlc4LuE46uoCI5CmaIeyUmoI0oAQ1LxxC/YYL/acMZ22f1HzBVurtnL+seeztqAaq0XIzRo2SLlUSqmBowEBMMbQtO0LopJbIDat076PSz4GYG7OXNbkVzMxPY7oCF0CSik19GhAADwlJfhqG4lO9kB0cqd9K0tXckz8MSRFprKxuEbnHyilhiwNCEDzunUARGVHg7X97t/ldbG2bC0z0mewtbQOl8evAUEpNWRpQACa1q3D4rDgyEzttH1d+TrcPjenpJ/C2oJqQCekKaWGLg0IQPO69UQNtyFxnfsPVpWuwm6xMzVtKmsL9pMxLIrh8ZGDlEullBpYYR8QfPX1uHfvJirF061DeWXpSqakTiHKFsWa/GryRmpzkVJq6Ar7gNC8YSMYQ3R8DcS2NxlVNFWwq3oXM9JnUFzdTHm9W/sPlFJDmgaE9etzbNQCAAAgAElEQVTAYiFyWFOnGsInez8B6NR/oAFBKTWUhX1AaFq3HscxI7HaTaeAsLJ0JYmRiRyXeBxrCvYT67AxbnjcIOZUKaUGVlgHBOP10rxpE9HjsgMbgk1GfuNnVekqTh5xMhaxsCa/mhOzh2HVBe2UUkNYWAcE186dmKYmoo4JPiUtWEPYVb2LKlcVp6SfQp3Lw86yeqZka3ORUmpoC+uA0LxuPQDRmTGBDcEawsrSlQDMSJ/BxqIajNH+A6XU0BfeAWH9OmzDh2N3NIE1AiIDi9atLF3JscOOJTU6lU3FtQCckKkL2imlhrawDghN6zcQPeVEaCgPNBeJ0OxtZl3ZOmamzwRgc3EtOUnRxEfbBzm3Sik1sMI2IHj27sW7dy9RJ06BhrK25qK1ZWvx+D2ckn4KAJtLapmUET+YWVVKqcMibANCU+uCdid2qCEQaC6KsEQwJW0KlQ1uSmqamZypAUEpNfSFbUBwbdmKOBxEjjuuUw1hVekqpqZNJdIWyeaSQP/BpAztP1BKDX1hGxC85eXY0tIQi0BTJcSmUdZYxp6aPe3NRcEO5eMzdEKaUmroC9+AUFmJLSkJGivB+CE2ldVlqwE4Of1kADYV1zI6JQZnpHYoK6WGvvAOCMnJgeYigNg0ShtKARgVPwqAzSU1OtxUKRU2wjYg+CorsaUkBzqUAWLTqGiqwBnhxGF1UFbnoqzOrSOMlFJhIywDgr+lBV9tLdakpA41hFQqmitIjQp0Lrf2H+gII6VUuAjLgOCrqgLAlpzSHhBiAgEhOToZgE0ltVgEJqRrh7JSKjyEZUDwVgYDQmuTUYQTIqKpbKokJSqw0N3m4hrGpDqJjrANZlaVUuqwCdOAUAEQGGUUnINgjKGiuYKU6BSMMYEZytpcpJQKIyEFBBE5W0R2isgeEbmxh/05IrJURDaJyIcikhncfoaIbOjw4xKRC4L7HhORLzvsy+3fS/tq3spKgOAoo8As5Vp3LR6/h5SoFPbWuqhsaNH+A6VUWDloQBARK/A34BxgAnCpiEzokmwJ8IQxZjKwCLgbwBjzgTEm1xiTC8wBmoB3Ohx3fet+Y8yGQ7+c0LT2IVhbh50GO5QBUqJS2lY41RFGSqlwEkoNYRqwxxjzhTGmBXgGOL9LmgnA0uDrD3rYD3Ax8KYxpqmvme0v3opKLE4nFoejrYbQFhCiU9hcUoPNIowfoR3KSqnwEUpAyACKOrwvDm7raCNwUfD1fMApIkld0nwbeLrLtsXBZqZ7RcTR04eLyDUiskZE1lRUVISQ3YNrm5TmaQZ3baCG0NS5hjA2zUmk3dovn6eUUkeDUAJCTw8SNl3eXwfMFpH1wGygBPC2nUBkBDAJeLvDMQuBccBJQCJwQ08fbox50BiTZ4zJS0lJCSG7B+etqmzvP4BONYSkyCQ2l9Rq/4FSKuyEEhCKgawO7zOB0o4JjDGlxpgLjTEnAjcHt9V2SLIAeMkY4+lwzF4T4AYeJdA0dVj4KiqxJid1CgiVzZXE2mOpqhdqmjw6wkgpFXZCCQirgTEiMkpEIgg0/bzaMYGIJItI67kWAo90OceldGkuCtYaEBEBLgC29D77fRNoMkrpNEu5vKmc5KhkNpXUADBZl7xWSoWZgwYEY4wX+DmB5p7twHPGmK0iskhEvhlMdjqwU0R2AWnA4tbjRWQkgRrGR11O/aSIbAY2A8nAnYd0JSHyNzfjb2xsn4MAbTWElOgUNhfXEmG1MHZ47OHIjlJKHTFCmoZrjHkDeKPLtv/p8Pp54PmvODaf7p3QGGPm9Caj/cVb1XGW8i5AICaZiqYKJqVMYtOOWsaNcOKwaYeyUiq8hN1MZW9wpFLb0tfRSRiLLTBLOSqFLfoMZaVUmAq7gNA2KS2pfZZyvacet8+N1R9PvdurI4yUUmEp7AJC27IVKe2zlCubAtsam6MBfYayUio8hV9AqAgGhMREaOw8S9njDnQkZyVGDVr+lFJqsIRfQKiqxDpsGGKzBZuMAkNOAbwtTuxWIdahS14rpcJP2JV83tZHZ7rrwOtqG3IK0OyKJjHGEJgaoZRS4SXsagiBWcrdl62IskVR12QjITpicDOolFKDJOwCgreqCltScudnKTcFhpxWN7aQFKsBQSkVnsIqIBhj2lc67TBLuaK5guSoZKqbPFpDUEqFrbAKCP7GJozL1f4sZQgMO22uJDU6laoGN0kxGhCUUuEprAKCL/gsZWvrOkYWO0QlUN5UTmJkMnUuLwkaEJRSYSqsAkL7s5RToL4MYtNo9DbR7G0mxpYAoDUEpVTYCrOA0HFhu33gTGt7UlqUBAKC1hCUUuEqzAJCsIaQlBSsIQxvm6VsNYH1ixI1ICilwlSYBYQKsFiwJiR0qyHgjQM0ICilwldYBQRfVRXWpETE+KCpqlMNwed1AhoQlFLhK6wCgreiMjAprTE45NQZWLbCYXXQ2GQH0HkISqmwFV4BoXVSWn3rpLThbc9Srmn2EBdpw24Nq69EKaXahFXp562qCs5S3hfYEKwhpESlUNXYos1FSqmwFjYBoX3ZiiSoDwaEYB9CSnRgHSMNCEqpcBY2AcFfWwseT3Cl0zJAICalbWE7rSEopcJd2AQEb/BZyoFZyvsgJoVm46HB06A1BKWUIpwCQuujM5OD6xjFprU9Szk5Kpn9jS06S1kpFdbCJyC0rWOUHKghONMobw4MP3Xakmjx+XUdI6VUWAubgOCr6hAQGjovW2EPLluhcxCUUuEsbAKCt7IS7HYsztjAsxCc7U1GEgwI+rQ0pVQ4C5+AUFGJLSkJad4PxtdWQ7Bb7LS4HYDWEJRS4S2kgCAiZ4vIThHZIyI39rA/R0SWisgmEflQRDKD288QkQ0dflwickFw3ygR+VREdovIsyIyoKVx26S0+vZJaRVNrY/O9AKQFOMYyCwopdQR7aABQUSswN+Ac4AJwKUiMqFLsiXAE8aYycAi4G4AY8wHxphcY0wuMAdoAt4JHnMPcK8xZgxQDVzVD9fzlbyVgRpC+7OU2yel7W90A5AQYx/ILCil1BEtlBrCNGCPMeYLY0wL8Axwfpc0E4Clwdcf9LAf4GLgTWNMk4gIgQDxfHDf48AFvc18b3grK7CmdK4htC5bsb/Rg90qxDpsA5kFpZQ6ooUSEDKAog7vi4PbOtoIXBR8PR9wikhSlzTfBp4Ovk4Caowx3gOcEwARuUZE1ojImoqKihCy253x+fDtr+68jlGHhe32N7pJjIkgEKeUUio8hRIQeiolTZf31wGzRWQ9MBsoAVoLe0RkBDAJeLsX5wxsNOZBY0yeMSYvJSUlhOx256upAZ8vsPR1fRlExuO2CHUtdaRGp7K/0aMdykqpsBdKG0kxkNXhfSZQ2jGBMaYUuBBARGKBi4wxtR2SLABeMsZ4gu8rgWEiYgvWErqdsz+1TUpLSYbafRA7nMrmwLaUqBSqm1p0yKlSKuyFUkNYDYwJjgqKIND082rHBCKSLCKt51oIPNLlHJfS3lyEMcYQ6Gu4OLjpe8Arvc9+aDrPUi7r9OjMtmUrtIaglApzBw0IwTv4nxNo7tkOPGeM2Soii0Tkm8FkpwM7RWQXkAYsbj1eREYSqGF81OXUNwC/EZE9BPoUHj6kKzkAXzAgWJOSAn0IHWYpB5qMWnTZCqVU2AtpWI0x5g3gjS7b/qfD6+dpHzHU9dh8eugwNsZ8QWAE04A7UA1hmCOR2uY9urCdUgPA4/FQXFyMy+Ua7KyEhcjISDIzM7Hb+zaEPizGWXorq5DISCxWD/jcbX0IVrGCLxZAawhKDYDi4mKcTicjR47UUXwDzBhDVVUVxcXFjBo1qk/nCIulK1onpUlDYHVTnMNp8DQQY4+hJjhLWWsISvU/l8tFUlKSBoPDQERISko6pNpYWAQEX1VllzkIabh9biKtkexvbAHQh+MoNUA0GBw+h/pdh0WTUdY//4m/uRk+/29gg3M4riIXDptDA4JSSgWFRQ1BrFassbHdawi2SPY3aUBQKtxcffXVbNu2bUA/49xzz6Wmpqbb9ttuu40lS5YM6Gf3VVjUENrUl4E9GhxOXD5XoMmoIRAQdB6CUuHjoYceGvDPeOONNw6e6AgTXgGhYR/EpoEIbq8bh9VBdVMLcZE27NawqCwpNWhuf20r20rr+vWcE9LjuHXexAOmaWxsZMGCBRQXF+Pz+bjlllt44IEHWLJkCXl5eTz88MPcc889pKenM2bMGBwOB/fffz9XXnklUVFR7Nixg4KCAh599FEef/xxVq1axfTp03nssccAePrpp7nrrrswxnDeeedxzz33ADBy5EjWrFlDcnIyixcv5oknniArK4uUlBSmTp3ar99DfwmvUrC+LBAQALfPjcPmoKqxRZuLlBrC3nrrLdLT09m4cSNbtmzh7LPPbttXWlrKHXfcwSeffMK7777Ljh07Oh1bXV3N+++/z7333su8efO49tpr2bp1K5s3b2bDhg2UlpZyww038P7777NhwwZWr17Nyy+/3Okca9eu5ZlnnmH9+vW8+OKLrF69+rBcd1+EXw0hLXA34fK5SLWmUqkBQanD4mB38gNl0qRJXHfdddxwww184xvf4NRTT23b99lnnzF79mwSExMB+Na3vsWuXbva9s+bNw8RYdKkSaSlpTFp0iQAJk6cSH5+PgUFBZx++um0Lrz5ne98h2XLlnHBBe2r+X/88cfMnz+f6OhoAL75zW9ypArDGsJwAFxeFw6r1hCUGurGjh3L2rVrmTRpEgsXLmTRokVt+wLLqn01hyPwFEWLxdL2uvW91+s96PGtjpaht+ETEFoaoaUenMEmI29glFG1BgSlhrTS0lKio6O5/PLLue6661i3bl3bvmnTpvHRRx9RXV2N1+vlhRde6NW5p0+fzkcffURlZSU+n4+nn36a2bNnd0pz2mmn8dJLL9Hc3Ex9fT2vvfZav1zXQAifJqP69gfjQKDJyGEJzEPQWcpKDV2bN2/m+uuvx2KxYLfbeeCBB7juuusAyMjI4KabbmL69Omkp6czYcIE4uPjQz73iBEjuPvuuznjjDMwxnDuuedy/vmdHxg5ZcoULrnkEnJzc8nJyenUZHWkkVCrPEeCvLw8s2bNmr4dXLASHj0HLn8Bjv0aef/O4+Ix3+YfL43npnPHcc1px/RvZpVSbN++nfHjxw92Ng6ooaGB2NhYvF4v8+fP5wc/+AHz588f7Gz1WU/fuYisNcbkHezY8GkyaigL/Bs7HGMMbp8b4w9UkHQOglLh67bbbiM3N5fjjz+eUaNGdeoQDjdh1GQUDAjO4bh9bgC83sDl69PSlApfR+qs4cEQRjWEfWCxQVRie0DwWQGtISilFIRTQGidlGax4PIGlodt8QQCQlKM40BHKqVUWAifgNC6bAW01RDcLcEaQkzfni6klFJDSfgEhPoycLYPOQVwtViIsFqIdYRPV4pSSn2V8AkIHWsI3kANodltISHGftTMIlRK9V5NTQ1///vf+3z86aefTp+Hux9lwiMgeFugqapbDaHBLSRq/4FSQ9qhBoRwEh5tJY3BZyl36UNodAmJ2n+g1OHx5o2wb3P/nnP4JDjn9wdMcuONN/L555+Tm5vLGWecwaZNm6iursbj8XDnnXdy/vnnk5+fzznnnMOsWbNYuXIlGRkZvPLKK0RFRQHwn//8h5/+9KfU1NTw8MMPH9GzjQ9FeNQQOsxBgPYmo/omrSEoNdT9/ve/55hjjmHDhg388Y9/5KWXXmLdunV88MEH/Pa3v21boG737t387Gc/Y+vWrQwbNqzTukZer5fPPvuMv/zlL9x+++2DdSkDLjxqCB0enQntTUa1TZAYrTUEpQ6Lg9zJHw7GGG666SaWLVuGxWKhpKSEsrLADeOoUaPIzc0FYOrUqeTn57cdd+GFF/a4fagJj4DQurBdaw0h2GRU3yy6sJ1SYeTJJ5+koqKCtWvXYrfbGTlyJC5X4Aax4/LWVquV5ubmtvet+6xWK16v9/Bm+jAKjyajhjJAICYVoG1iGsZGkgYEpYY0p9NJfX09ALW1taSmpmK32/nggw8oKCgY5NwdWcKnhhCTDNbA5bbWEIzfrjUEpYa4pKQkZs6cyfHHH89JJ53Ejh07yMvLIzc3l3Hjxg129o4oIQUEETkbuA+wAg8ZY37fZX8O8AiQAuwHLjfGFAf3ZQMPAVmAAc41xuSLyGPAbKA2eJorjTEbDvmKetLQ/qQ0aO9DwNj14ThKhYGnnnrqoGm2bNnS9rr1eQkAH374Ydvr5OTk8O5DEBEr8DdgLlAMrBaRV40x2zokWwI8YYx5XETmAHcDVwT3PQEsNsa8KyKxgL/DcdcbY57vjws5oJOuhpaGtrdurxur2ACLBgSllAoKpYYwDdhjjPkCQESeAc4HOgaECcC1wdcfAC8H004AbMaYdwGMMQ0MhjFzO711+9xYJRAINCAopVRAKJ3KGUBRh/fFwW0dbQQuCr6eDzhFJAkYC9SIyIsisl5E/hiscbRaLCKbROReETlsEwJcPhdWAoFAl75WSqmAUAJCTwv9dH3u5nXAbBFZT6BfoATwEqiBnBrcfxIwGrgyeMxCYFxweyJwQ48fLnKNiKwRkTUVFRUhZPfg3F43FuzEOmzYreEx0EoppQ4mlNKwmECHcKtMoLRjAmNMqTHmQmPMicDNwW21wWPXG2O+MMZ4CTQlTQnu32sC3MCjBJqmujHGPGiMyTPG5KWkpPTy8nrm8rkQIoi0azBQSqlWoZSIq4ExIjJKRCKAbwOvdkwgIski0nquhQRGHLUemyAirSX5HIJ9DyIyIvivABcA7V38A8ztc2MxdiK0dqCUUm0OWiIG7+x/DrwNbAeeM8ZsFZFFIvLNYLLTgZ0isgtIAxYHj/URaC5aKiKbCTQ//TN4zJPBbZuBZODOfruqg3B73Qh2HHbrwRMrpYacq6++mm3bth084WFw8803k5WVRWxsbLd9e/fu5ayzzmLDhg3MmDGDiRMnMnnyZJ599tkByUtI8xCMMW8Ab3TZ9j8dXj8P9Dh8NDjCaHIP2+f0Kqf9yOVzgbHjsGkNQalw9NBDDw12FtrMmzePn//854wZM6bbvrfeeouvf/3rREdH88QTTzBmzBhKS0uZOnUqX//61xk2bFi/5iU8Zip34fa5wTg0ICh1GN3z2T3s2L+jX885LnEcN0zrcTwKAI2NjSxYsIDi4mJ8Ph+33HILl1xyCaeffjpLliwhLy+Phx9+mHvuuYf09HTGjBmDw+Hg/vvv58orryQqKoodO3ZQUFDAo48+yuOPP86qVauYPn06jz32GAA/+clPWL16Nc3NzVx88cW9Xg315JNP/sp9b731Frfeeitjx45t25aenk5qaioVFRUaEPqDy+vC+GNx2LTJSKmh7K233iI9PZ3//ve/QGAto45KS0u54447WLduHU6nkzlz5nDCCSe07a+urub999/n1VdfZd68eaxYsYKHHnqIk046iQ0bNpCbm8vixYtJTEzE5/Nx5plnsmnTJiZP7tYo0ms+n4+dO3cyYcKETts/++wzWlpaOOaYYw75M7oKy4AQqCHYcegoI6UOmwPdyQ+USZMmcd1113HDDTfwjW98o9uDbT777DNmz55NYmIiAN/61rfYtWtX2/558+YhIkyaNIm0tDQmTZoEwMSJE8nPzyc3N5fnnnuOBx98EK/Xy969e9m2bVu/BIRPP/2U6dOnd9q2d+9errjiCh5//HEslv4vv8KyRHT73Pj9Nh1lpNQQN3bsWNauXcukSZNYuHAhixYt6rS/9eE4X6V12WuLxdJpeWyLxYLX6+XLL79kyZIlLF26lE2bNnHeeee1LafdqqioiNzcXHJzc/nHP/4Rct7ffPNNzj777Lb3dXV1nHfeedx5550HbGY6FGFZIrq8Lvw+m9YQlBriSktLiY6O5vLLL+e6665j3bp1nfZPmzaNjz76iOrqarxeb6enpIWirq6OmJgY4uPjKSsr48033+yWJisriw0bNrBhwwZ+/OMfh3zupUuXcuaZZwLQ0tLC/Pnz+e53v8u3vvWtXuWxN8K2ycjus2sfglJD3ObNm7n++uuxWCzY7XYeeOCBTvszMjK46aabmD59Ounp6UyYMIH4+PiQz3/CCSdw4oknMnHiREaPHs3MmTN7ncff/e53PPXUUzQ1NZGZmcnVV1/Nz372MyIjI4mLiwPgueeeY9myZVRVVbV1Zj/22GNtT3jrL3KwKtORJC8vz6xZs+aQzuHxe5jyrynY687hrPTL+f1Fh97Wp5Tq2fbt2xk/fvxgZ+OAGhoaiI2Nxev1Mn/+fH7wgx8wf/78Qc3Tv//9b4qLi7nxxht7fWxP37mIrDXG5B3s2LCrIbi9gYfjeL02HXaqlOK2227jvffew+VycdZZZ3HBBRcMdpa4/PLLB+Vzwy4gtD4cx+ezEaEBQamwt2TJksHOwhEj7ErE1sdnerxW7UNQSqkOwi8gBJuM/H5dukIppToKuxKx7XnKfh12qpRSHYVdidjaZGSMDjtVSqmOwi4guLytNQRtMlJqqCsqKuKMM85g/PjxTJw4kfvuuy/kY9esWcMvf/nLAcxd6Hbs2MGMGTNwOBwD2gkedqOMOtYQdJSRUkObzWbjT3/6E1OmTKG+vp6pU6cyd+7cbgvG9SQvL4+8vIMO3T8sEhMT+etf/8rLL788oJ8TdgGhvQ9Bm4yUOpz23XUX7u39u/y1Y/w4ht9001fuHzFiBCNGjADA6XQyfvx4SkpKugWE//znP9x+++1YrVbi4+NZtmwZH374IUuWLOH111+noqKCyy67jKqqKk466STeeust1q5dS0NDA2effTazZs3ik08+4YQTTuD73/8+t956K+Xl5Tz55JNMmzaNzz77jF//+tc0NzcTFRXFo48+ynHHHRfydaamppKamtq2autACbtb5NZRRsboxDSlwkl+fj7r16/vtoIowKJFi3j77bfZuHEjr776arf9t99+O3PmzGHdunXMnz+fwsLCtn179uzhV7/6FZs2bWLHjh089dRTLF++nCVLlnDXXXcBMG7cOJYtW8b69etZtGgRNx0giA2msKshtDYZ6fLXSh1eB7qTH2gNDQ1cdNFF/OUvf2lbH6ijmTNncuWVV7JgwQIuvPDCbvuXL1/OSy+9BMDZZ59NQkJC275Ro0Z1Whb7zDPPbFsyOz8/Hwg8h+F73/seu3fvRkTweDwDcJWHLuxKxNZOZaNNRkqFBY/Hw0UXXcR3vvOdHgt7gH/84x/ceeedbUtVV1VVddp/oDXfui6L3XHJbK/XC8Att9zCGWecwZYtW3jttde6LZENgWcrty6TPVjCLiB0rCFop7JSQ5sxhquuuorx48fzm9/85ivTff7550yfPp1FixaRnJxMUVFRp/2zZs3iueeeA+Cdd96hurq6V/mora0lIyMDoG210q4WL17ctkz2YAm7EtHlcyEIGKv2ISg1xK1YsYJ//etfvP/++21332+88Ua3dNdffz2TJk3i+OOP57TTTuv0GE2AW2+9lXfeeYcpU6bw5ptvMmLECJxOZ8j5+N3vfsfChQuZOXMmPp+v19exb98+MjMz+fOf/8ydd95JZmYmdXV1vT7PwYTd8tdLVi/hqR3PsH/r7bz/29mMTontp9wppbo6Gpa/DoXb7cZqtWKz2Vi1ahU/+clPBvVO/kB0+etecPlc2CTQxuewax+CUurgCgsLWbBgAX6/n4iICP75z38OdpYGRNgFBLfPjU0iALTJSCkVkjFjxrB+/frBzsaAC7sS0e11Y9WAoNRhczQ1Sx/tDvW7DrsS0eVzYSUQEHSUkVIDKzIykqqqKg0Kh4ExhqqqKiIjI/t8jrBsMrJgByDCqgFBqYGUmZlJcXExFRUVg52VsBAZGUlmZmafjw+7gODyuhAicNgsiMhgZ0epIc1utzNq1KjBzoYKUUi3yCJytojsFJE9InJjD/tzRGSpiGwSkQ9FJLPDvmwReUdEtovINhEZGdw+SkQ+FZHdIvKsSLBhf4C5fW7E6NLXSinV1UFLRRGxAn8DzgEmAJeKSNe1Y5cATxhjJgOLgLs77HsC+KMxZjwwDSgPbr8HuNcYMwaoBq46lAsJldvnRrDrkFOllOoilNvkacAeY8wXxpgW4Bng/C5pJgBLg68/aN0fDBw2Y8y7AMaYBmNMkwTaauYAzwePeRy44JCuJEQuryuwbIX2HyilVCeh9CFkAB0X9igGuq4fuxG4CLgPmA84RSQJGAvUiMiLwCjgPeBGIAGoMcZ4O5wzo6cPF5FrgGuCbxtEZGcIee5JMlDZ/vZBZGEfz3Rk6HI9R7WhdC0wtK5nKF0LhO/15IRyslACQk89r13HkF0H3C8iVwLLgBLAGzz/qcCJQCHwLHAl0H3B8e7nDGw05kHgwRDyeUAisiaUqdtHi6F0PUPpWmBoXc9QuhbQ6zmYUNpNioGsDu8zgdKOCYwxpcaYC40xJwI3B7fVBo9dH2xu8gIvA1MIRLRhImL7qnMqpZQ6vEIJCKuBMcFRQRHAt+lyhy8iySLSeq6FwCMdjk0QkZTg+znANhOYpfIBcHFw+/eA/9/e2bzIUUVR/HdQ8SMqMVHDgELIRt2YiQtJiEiMH4Qg+QdcuowQUVAHQVDIVnTlxqgbETR+RGZhHFrdJiRmYmaMEyMOGIyOLkRwIRqvi3cbmjjYMz09U/26zw+Kqne7Gu6hX3Grb70+faR3GcYYY1ZK14KQd/ZPAEeBs8C7ETEr6SVJ+/K0XcCcpHPAJuBgvvcSpZ3UknSG0n5qu0I9Czwl6TywETjUN1WLs+K204AxTHqGSQsMl55h0gLW879UZX9tjDFm9fDaS2OMMYALgjHGmGQkCkI3641BRtIbkhYkzXTENkiaStuPKUk3NZnjcpB0u6TP08pkVtKBjFenSdI1ko5LOp1aXsx4I7Ys/ULSFZJOSbqREUYAAAK+SURBVJrMcZV6JM1LOiNpWtKJjFU3z9pIWi/psKRv8vrZ0W89Q18Qlmi9Mci8Bey5LPYc0Erbj1aOa+Fv4Om0MtkO7M/Po0ZNfwK7I2IrMA7skbSdhmxZ+sgBygKSNjXreSAixjvW6tc4z9q8CnwSEXcCWymfUX/1RMRQb8AO4GjHeAKYaDqvZWrYDMx0jOeAsTweA+aaznEF2o4AD9euCbgO+JLyK/5fKZYt/5l/g75RfhPUoiwRn6SsDKxSDzAP3HxZrMp5BtwIfE8uBFotPUP/DYHFrTcWtcmoiE0RcREg97c2nE9PpPPtNuAYlWrK9so0xbRxCviOJdqyDCivAM8A/+R4I/XqCeBTSSfTAgcqnWfAFuAX4M1s570uaR191jMKBWEp1htmjZF0PfA+8GRE/N50Pr0SEZciYpxyZ30vcNdip61tVr0h6VFgISJOdoYXObUKPcDOiLiH0i7eL+n+phNaAVdSXB5ei+II8Qer0O4ahYLQ1XqjQn6WNAaQ+4Uu5w8Ukq6iFIO3I+KDDFetKSJ+A76gPBep1ZZlJ7BP0jzF1Xg35RtDlXoi4sfcLwAfUgp2rfPsAnAhIo7l+DClQPRVzygUhK7WGxXyMcXuAyqz/Ujr80PA2Yh4ueOl6jRJukXS+jy+FniI8qCvSluWiJiIiNsiYjPlOvksIh6jQj2S1km6oX0MPALMUOE8A4iIn4AfJN2RoQeBr+m3nqYflqzRA5m9wDlKf/f5pvNZZu7vABeBvyh3CY9T+rot4Nvcb2g6z2XouY/ScvgKmM5tb42agLuBU6llBngh41uA48B54D3g6qZz7UHbLmCyVj2Z8+ncZtvXfY3zrEPTOHAi59tHlL8R6KseW1cYY4wBRqNlZIwxZgm4IBhjjAFcEIwxxiQuCMYYYwAXBGOMMYkLgjHGGMAFwRhjTPIvH51klMKor2MAAAAASUVORK5CYII=\n",
      "text/plain": [
       "<Figure size 432x288 with 1 Axes>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "accuracies_shifted_2sig = [\n",
    "    0.9689, 0.9766, 0.9797, 0.9796, 0.9818, 0.982, 0.9831, 0.9838, 0.9845, 0.9844, 0.9844, 0.9849, 0.9847, 0.985, 0.985,\n",
    "    0.9852, 0.9853, 0.9854, 0.9856, 0.9854, 0.9856, 0.9855, 0.9854, 0.9854, 0.9854, 0.9856, 0.9856, 0.9856, 0.9856,\n",
    "    0.9856, 0.9856, 0.9856, 0.9856, 0.9856, 0.9855, 0.9855, 0.9855, 0.9856, 0.9856, 0.9856, 0.9856, 0.9856, 0.9856,\n",
    "    0.9857, 0.9857, 0.9857, 0.9857, 0.9857, 0.9857, 0.9857, 0.9857, 0.9858, 0.9859, 0.9859, 0.9859, 0.9859, 0.9859,\n",
    "    0.9859, 0.9859, 0.9859\n",
    "]\n",
    "\n",
    "plt.ylim(bottom=0.96, top=0.99)\n",
    "plt.plot(accuracies_sigmoid, label=\"sigmoid\")\n",
    "plt.plot(accuracies_tanh, label=\"tanh\")\n",
    "plt.plot(accuracies_shifted_sig, label=\"sigma - 1/2\")\n",
    "plt.plot(accuracies_shifted_2sig, label=\"2 sigma - 1\")\n",
    "plt.legend()\n",
    "plt.show()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "The only thing that is improved is the classification accuracy of the very first epochs. Then our stabilized accuracies are worse.\n",
    "\n",
    "With the caveats that I only tested each configuration once and didn't optimize the hyper-parameters every time, this seems to suggest that the \"*symmetric about the origin*\" argument was more relevant than the \"*usually higher derivatives*\" one.\n",
    "\n",
    "Due to the execution time on my CPU, I didn't\n",
    "> Try a half-dozen iterations on the learning hyper-parameters or network architecture, searching for ways that tanh may be superior to the sigmoid.\n",
    "\n",
    "But you shouldn't have any trouble doing it."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Problem 3 ([link](http://neuralnetworksanddeeplearning.com/chap6.html#problem_437600))\n",
    "\n",
    "By displacing each training image by a single pixel (up, down, left, or right), the convolutional layers are only displaced by a single pixel, too. But then the max-pooling layer can look quite different, as for each of its neurons, the 4 activations taken into account for the computation of the maximum will be different (more precisely, 2 will remain, but 2 will be new)."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## The code for our convolutional networks"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Problems 4 - 9 ([link](http://neuralnetworksanddeeplearning.com/chap6.html#problems_269956)): some improvements of `network3.py`\n",
    "\n",
    "I'll consider each of these problems in detail below, but a few remarks apply to all of them:\n",
    "* the code is in the `chap6p4-9` directory;\n",
    "* because running the most complex networks would take too much time on my CPU, I've trained a simple network in each of the problems below: 784 input neurons, one ConvPoolLayer with 5 feature maps, a single fully-connected layer with 30 neurons, and a softmax output layer (see `exec_network3.py`)."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Problem 4: implement early stopping\n",
    "\n",
    "As in Chapter 3, Problem 12, I've implemented a no-improvement-in-$n$-epochs early stopping schedule.\n",
    "\n",
    "In `network3p4.py`:\n",
    "* I've tracked the past validation accuracies in the variable `past_accuracies`;\n",
    "* the SGD method doesn't take `epochs` as a parameter anymore, but `es` instead (for *early stopping*, the $n$ in \"no-improvement-in-$n$-epochs);\n",
    "* the main loop is not:\n",
    "\n",
    "```\n",
    "for epoch in range(epochs):\n",
    "```\n",
    "\n",
    "anymore, but:\n",
    "\n",
    "```\n",
    "while (len(past_accuracies) <= es or\n",
    "       max(past_accuracies[-es:]) > past_accuracies[-es - 1]):\n",
    "```\n",
    "\n",
    "And the variable `epoch` is incremented manually at each iteration.\n",
    "\n",
    "Testing (`exec_p4.py`) with `es=2` we get:\n",
    "\n",
    "```\n",
    "Running with a CPU.  If this is not desired, then the modify network3.py to set\n",
    "the GPU flag to True.\n",
    "...\n",
    "Epoch 0: validation accuracy 93.40%\n",
    "...\n",
    "Epoch 10: validation accuracy 98.22%\n",
    "...\n",
    "Epoch 11: validation accuracy 98.22%\n",
    "...\n",
    "Epoch 12: validation accuracy 98.21%\n",
    "Finished training network.\n",
    "Best validation accuracy of 98.22% obtained at iteration 59999\n",
    "Corresponding test accuracy of 98.00%\n",
    "```"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Problem 5: Add a `Network` method to return the accuracy on an arbitrary data set\n",
    "\n",
    "TODO"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Problem 6: Modify the `SGD` method to allow the learning rate $\\eta$ to be a function of the epoch number\n",
    "\n",
    "We'll implement an exponential decrease in the learning rate (which is the easiest to implement, and a reasonable schedule). I'd like the learning rate to be divided by 2 every 10 epochs. So we need to multiply it by $\\frac{1}{2^{1/10}}$ at every epoch.\n",
    "\n",
    "It's more convenient to update the learning rate with the other parameters, for every mini-batch. Since there are 5,000 mini-batches per epoch (with a mini-batch size of 10), this means that we'll want to multiply the learning rate by $\\frac{1}{2^{1/50,000}} \\approx 0.999986$ for every mini-batch.\n",
    "\n",
    "The notable changes in `network3p6.py` are:\n",
    "\n",
    "* I've created a shared variable for the learning rate, in order to update it easily using Theano:\n",
    "\n",
    "\n",
    "```\n",
    "eta_shared = theano.shared(np.array(eta, dtype=theano.config.floatX))\n",
    "```\n",
    "\n",
    "* I've updated it using the list `updates`:\n",
    "\n",
    "```\n",
    "updates = [(param, param-eta_shared.get_value()*grad)\n",
    "           for param, grad in zip(self.params, grads)]\n",
    "updates.append((eta_shared, 0.999986 * eta_shared))\n",
    "```\n",
    "\n",
    "Executing `exec_p6.py` gives:\n",
    "\n",
    "```\n",
    "Running with a CPU.  If this is not desired, then the modify network3.py to set\n",
    "the GPU flag to True.\n",
    "...\n",
    "Epoch 0: validation accuracy 93.47%\n",
    "This is the best validation accuracy to date.\n",
    "The corresponding test accuracy is 92.87%\n",
    "New learning rate: 0.093\n",
    "...\n",
    "Epoch 9: validation accuracy 98.07%\n",
    "This is the best validation accuracy to date.\n",
    "The corresponding test accuracy is 97.85%\n",
    "New learning rate: 0.050\n",
    "...\n",
    "Epoch 19: validation accuracy 98.43%\n",
    "This is the best validation accuracy to date.\n",
    "The corresponding test accuracy is 98.08%\n",
    "New learning rate: 0.025\n",
    "```\n",
    "\n",
    "Which is what we want."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Problem 7: expand the training data with rotations, skewing, and translation\n",
    "\n",
    "When Nielsen expanded the data set with translations of a single pixel, it multiplied the size of the training data by 5. To incorporate these 3 techniques, we won't be able to explicitly generate the expanded data set. Instead, for every mini-batch, we'll randomly transform each of its 10 images by combining:\n",
    "\n",
    "* a random rotation chosen uniformly between -5 and 5 degrees;\n",
    "* a random translation of -1, 0, or 1 pixel (with equal probabilities) horizontally;\n",
    "* the same vertically;\n",
    "* a random skewing (centered in the middle of the image), defined by a translation of the upper pixels of -1, 0, or 1 pixel.\n",
    "\n",
    "I've written `display_transformed_image.py` to test those changes. With the first picture in MNIST:\n",
    "\n",
    "<img src=\"img/5.png\" alt=\"img/5.png\" style=\"width: 300px;\"/>\n",
    "\n",
    "We arrive at pictures like this:\n",
    "\n",
    "<div style=\"display: table;\">\n",
    "    <div style=\"display: table-cell; vertical-align: middle;\">\n",
    "        <img src=\"img/5_tr1.png\" alt=\"img/5.png\" style=\"width: 300px;\"/>\n",
    "    </div>\n",
    "    <div style=\"display: table-cell; vertical-align: middle;\">\n",
    "        <img src=\"img/5_tr2.png\" alt=\"img/5.png\" style=\"width: 300px;\"/>\n",
    "    </div>\n",
    "    <div style=\"display: table-cell; vertical-align: middle;\">\n",
    "        <img src=\"img/5_tr3.png\" alt=\"img/5.png\" style=\"width: 300px;\"/>\n",
    "    </div>\n",
    "    <div style=\"display: table-cell; vertical-align: middle;\">\n",
    "        <img src=\"img/5_tr4.png\" alt=\"img/5.png\" style=\"width: 300px;\"/>\n",
    "    </div>\n",
    "</div>\n",
    "\n",
    "Then I wasn't able to make it work with Theano :("
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Problem 8: Add the ability to load and save networks to `network3.py`\n",
    "\n",
    "I've added the following 2 methods to the `Network` class in `network3p8.py`:\n",
    "\n",
    "```\n",
    "def save(self, name):\n",
    "    \"\"\"Save the network at ``name``\"\"\"\n",
    "    with open(name, \"wb\") as f:\n",
    "        pickle.dump(self, f, protocol=pickle.HIGHEST_PROTOCOL)\n",
    "\n",
    "@staticmethod\n",
    "def load(name):\n",
    "    \"\"\"Return the network saved at ``name``\"\"\"\n",
    "    with open(name, \"rb\") as f:\n",
    "        return pickle.load(f)\n",
    "```\n",
    "\n",
    "Then executing `exec_p8_save.py` (which only trains the network for 1 epoch to save time) gives a standard output, followed by (the precise weight changes with each execution):\n",
    "\n",
    "```\n",
    "weight between the first neuron in the max-pooling layer and the first neuron in the fully-connected layer: 0.07260374367553253\n",
    "```\n",
    "\n",
    "Let's execute `exec_p8_load.py` and verify our networks are the same:\n",
    "\n",
    "```\n",
    "weight between the first neuron in the max-pooling layer and the first neuron in the fully-connected layer: 0.07260374367553253\n",
    "```"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Problem 9: add diagnostic tools to make it easier to understand to what extent a network is overfitting\n",
    "\n",
    "A key thing to do is compare the accuracies on the training set and the validation set over time.\n",
    "\n",
    "In `network3p9.py`, I've kept a list of the validation accuracies (as in problem 4), as well as a list of the training accuracies (the accuracies of the network on the whole training set, which obviously slows down learning substantially). I've also plotted those accuracies dynamically (with `plt.ion()`) during training.\n",
    "\n",
    "Training the network during 30 epochs, we finally get something like this (orange: training; blue: validation):\n",
    "\n",
    "![img/accuracies.png](img/accuracies.png)\n",
    "\n",
    "Our network (which, remember, is a very simple one for reasons of computation time) seems to be overfitting starting at epoch 4, when the accuracy on the training data becomes larger than the accuracy on the validation data."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Problem 10: weight initialization for rectified linear units\n",
    "\n",
    "Recall that $ReLU(z) = max(0, z)$. The property of the ReLU that we'll use is that for $c > 0$, $ReLU(cz) = c . ReLU(z)$. Because if $z \\leq 0$, then $cz \\leq 0$ and $ReLU(cz) = 0 = c . ReLU(z)$, and if $z > 0$, then $cz > 0$ and $ReLU(cz) = cz = c . ReLU(z)$.\n",
    "\n",
    "**This proof assumes that all biases are 0**. With non-null biases, the argument will still hold as an approximation.\n",
    "\n",
    "We consider a network of $L$ layers, indexed from $1$ (input layer) to $L$ (output layer).\n",
    "\n",
    "For the first hidden layer (layer 2), suppose we multiply all weights leading to this layer by a constant $c > 0$, i.e. we use $cw^2$ as our weight matrix.\n",
    "\n",
    "We'll call:\n",
    "* $a^1$ the activation of the input layer;\n",
    "* $a^2$ the activation of the first hidden layer without the multiplication by $c$;\n",
    "* $a'^2$ the activation of the first hidden layer, with the multiplication by $c$;\n",
    "* etc.\n",
    "\n",
    "All those activations are vectors. We then have:\n",
    "\n",
    "$$a'^2 = ReLU(cw^2.a^1) = c . ReLU(w^2 a^1) = c a^2$$\n",
    "\n",
    "The same reasoning applies to the second hidden layer:\n",
    "\n",
    "$$a'^3 = ReLU(cw^3.a'^2) = c . ReLU(w^3 a'^2) = c . ReLU(w^3 c a^2) = c^2 ReLU(w^3 a^2) = c^2 a^3$$\n",
    "\n",
    "And so on. We finally arrive to the output of our network:\n",
    "\n",
    "$$a'^{L} = c^{L-1} a^{L}$$\n",
    "\n",
    "So we've simply rescaled all outputs by a factor $c^{L-1}$.\n",
    "\n",
    "This doesn't change the classification returned by the network (because the neuron with the maximum output activation won't change). Neither the quadratic cost or the cross-entropy cost would be adapted to such a network made exclusively of ReLU neurons: the cross-entropy cost explicitly requires activations in the interval $]0,1[$, and the quadratic cost assumes that if $0 \\leq y \\leq 9$ is the correct classification on a certain input, then neuron $y$ in the final layer should have an output as close to 1 as possible, not some large value.\n",
    "\n",
    "Now if the final layer is a softmax, we have:\n",
    "\n",
    "$$a'^{L-1} = c^{L-2} a^{L-1}$$\n",
    "\n",
    "and so for the $j$th neuron in the final layer:\n",
    "\n",
    "$$z_j'^{L} = c^{L-1} z_j^{L}$$\n",
    "\n",
    "$$a_j'^{L} = \\frac{e^{z_j'^{L-1}}}{\\sum\\limits_k e^{z_k'^{L-1}}} = \\frac{e^{c^{L-1} z_j^{L-1}}}{\\sum\\limits_k e^{c^{L-1} z_k^{L-1}}}$$\n",
    "\n",
    "Which is the more general form of softmax that we saw in Chapter 3 Problem 6, with $c^{L-1}$ as the multiplying constant. So if $c > 1$, we tend toward a hardened version where the maximum activation is higher than without the multiplication, and all other activations are lower. In other words, the network is more confident in its classification.\n",
    "\n",
    "And if $c < 1$, we tend toward a softened version: all activations tend toward the same value and the network is less confident in its classification.\n",
    "\n",
    "Because of the power $L-1$, those two behaviors are exacerbated when the network has more layers.\n",
    "\n",
    "So compared to our first weight initialization (gaussian variables of mean 0 and standard deviation 1), the weight initialization as gaussian variables of mean 0 and standard deviation $\\frac{1}{\\sqrt{n_{in}}}$ makes the network less confident in its classification. Also, the reason behind this initialization (avoiding saturation) no longer holds. However, it's hard to tell which initialization is better, or to find a better initialization procedure. This is where the problem becomes \"*very open-ended*\", and I won't dive any deeper into it."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Problem 11: the unstable gradient problem with rectified linear units\n",
    "\n",
    "Recall the expression connecting the gradients in the toy network described in Chapter 5:\n",
    "\n",
    "$$\\frac{\\partial C}{\\partial b_1} = \\sigma'(z_1) w_2 \\sigma'(z_2) w_3 \\sigma'(z_3) w_4 \\sigma'(z_4) \\frac{\\partial C}{\\partial b_4}$$\n",
    "\n",
    "We generally had a vanishing gradient because we had $\\sigma'(z) \\leq \\frac 1 4$ for all $z$ and $\\sigma'$ could quickly take very low values as soon as $|z|$ increased.\n",
    "\n",
    "Now, the derivative of the ReLU function is:\n",
    "\n",
    "\\begin{eqnarray}\n",
    "  ReLU'(z) = \\left\\{ \n",
    "    \\begin{array}{ll} \n",
    "      0 & \\mbox{if } z \\leq 0 \\\\\n",
    "      1 & \\mbox{if } z > 0\n",
    "    \\end{array}\n",
    "  \\right.\n",
    "\\end{eqnarray}\n",
    "\n",
    "So the expression above would become:\n",
    "\n",
    "\\begin{eqnarray}\n",
    "  \\frac{\\partial C}{\\partial b_1} = \\left\\{ \n",
    "    \\begin{array}{ll} \n",
    "      0 & \\mbox{if at least one weighted input is negative} \\\\\n",
    "      w_2 w_3 w_4 \\frac{\\partial C}{\\partial b_4} & \\mbox{if all weighted inputs are non-negative}\n",
    "    \\end{array}\n",
    "  \\right.\n",
    "\\end{eqnarray}\n",
    "\n",
    "If all weighted inputs are non-negative, the gradient is still unstable because of the product of the weights. We can't do much to prevent this. However, the gradient is completely annihilated as soon as one weighted input is negative!\n",
    "\n",
    "Suppose one of the neurons learns a large negative bias at some point in training, while its weight vector has comparatively small components. Then it may happen that no input ever makes it output something positive. In this case, its gradient is always going to be 0, and its parameters won't change anymore. This doesn't look like something desirable! It's known as the \"dying ReLU\" problem.\n",
    "\n",
    "Some ideas to help address this problem:\n",
    "* avoiding high learning rates, since they make it more likely that a neuron learns a large negative bias at some point;\n",
    "* changing the activation function a little to always have a non-negative derivative:\n",
    "  * \"Leaky ReLU\":\n",
    "\\begin{eqnarray}\n",
    "  Leaky(z) = \\left\\{ \n",
    "    \\begin{array}{ll} \n",
    "      az & \\mbox{if } z < 0 \\\\\n",
    "      z & \\mbox{if } z \\geq 0\n",
    "    \\end{array}\n",
    "  \\right.\n",
    "\\end{eqnarray}\n",
    "With $a$ a small positive number.\n",
    "  * exponential linear units, ELU:\n",
    "\\begin{eqnarray}\n",
    "  ELU(z) = \\left\\{ \n",
    "    \\begin{array}{ll} \n",
    "      e^z - 1 & \\mbox{if } z < 0 \\\\\n",
    "      z & \\mbox{if } z \\geq 0\n",
    "    \\end{array}\n",
    "  \\right.\n",
    "\\end{eqnarray}\n",
    "  * one can think of many other activations functions satisfying this condition.\n",
    "\n",
    "As Nielsen writes: \"*The word good in the second part of this makes the problem a research problem*\", so I won't dive any deeper into these functions."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 6,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "image/png": "iVBORw0KGgoAAAANSUhEUgAAAXYAAAD8CAYAAABjAo9vAAAABHNCSVQICAgIfAhkiAAAAAlwSFlzAAALEgAACxIB0t1+/AAAADl0RVh0U29mdHdhcmUAbWF0cGxvdGxpYiB2ZXJzaW9uIDIuMi4yLCBodHRwOi8vbWF0cGxvdGxpYi5vcmcvhp/UCwAAIABJREFUeJzt3XuczPXix/HXZ2dv9uJOFyu2Uorksjq5VLqhCKVOKdfq8DvnxCKR6qCOrsqtdKRTuZZIUu5FpSutogiJ1C5hsVj2PvP5/bHaIxZ7mdnvzOz7+XjMY2a++53v9z1bvfvs9/udzxhrLSIiEjxCnA4gIiLepWIXEQkyKnYRkSCjYhcRCTIqdhGRIKNiFxEJMip2EZEgo2IXEQkyKnYRkSAT6sROq1evbuvWrevErkVEAtbatWv3WWtrnGk9R4q9bt26JCUlObFrEZGAZYz5tSjr6VCMiEiQUbGLiAQZFbuISJDxyjF2Y8wOIB1wA3nW2oTibiM3N5eUlBSysrK8EUmKIDIykri4OMLCwpyOIiJe5M2Tp9daa/eV9MUpKSnExsZSt25djDFejCWFsdayf/9+UlJSiI+PdzqOiHiR3xyKycrKolq1air1MmKMoVq1avoLSSQIeavYLbDcGLPWGNO3pBtRqZct/b5FgpO3ir2VtbYpcBPwT2PM1SeuYIzpa4xJMsYkpaamemm3IiKBIe1QKomv3kDy7u0+35dXit1au+vY/V5gPnBFIetMsdYmWGsTatQ44wenHOFyuWjcuDENGzbklltu4eDBg2d8TUxMzEnLevfuzTvvvHPG9USkfMjIOkr/2e35JGw3H3872+f7K3WxG2OijTGxfzwG2gIbSrtdJ1SoUIF169axYcMGqlatyqRJk5yOJCIBLi8vl8TpN7I+MofuEa3oefMjPt+nN0bsZwGfG2PWA2uARdbapV7YrqNatGjBzp07C56PGTOG5s2b06hRI0aOHOlgMhEJFB63myFv3MTXEencSn0e6vZKmey31Jc7Wmu3A5d7IUuBxz/YyI+7Dntzk1x6bkVG3tKgSOu63W5WrFjBfffdB8Dy5cvZunUra9aswVpLp06dWLVqFVdffdKpBBGRAiOm/5UV4Xu4IfccRvXx/SGYP/jN5Y7+IDMzk8aNG1OtWjUOHDjAjTfeCOQX+/Lly2nSpAlNmzZl8+bNbN269ZTbKexqE12BIlK+PPPm/SwI+YmW2ZUY03sRIS5Xme3bkdkdz6SoI2tv++MY+6FDh+jYsSOTJk1iwIABWGsZPnw4/fr1K9J2qlWrRlpaWsHzAwcOUL16dV/FFhE/8593h/Fmztc0zo5kXK9lhIaW7ae7NWIvRKVKlZg4cSLPP/88ubm5tGvXjtdff50jR44AsHPnTvbu3XvK17dp04a3336bnJwcAKZOncq1115bJtlFxFlvLhvDlMOLuDgnlBfvXkpUZHSZZ/DLEbs/aNKkCZdffjmzZ8+mR48ebNq0iRYtWgD5ly7OnDmTmjVrkpGRQVxcXMHrBg8ezODBg1m7di3NmjXD5XJxwQUXMHnyZKfeioiUkQ9Wvc74XVOJywthXJf3qBzrzF/qxlpb5jtNSEiwJ37RxqZNm7jkkkvKPEt5p9+7iHd8/t1Chn03jGiP4aUbZ3JRncZe34cxZm1RJlnUiF1EpJS+/+lLRnw7jFDgqRYv+aTUi0PFLiJSCtuTNzJsVV8yQ+DJho+T0KCN05F08lREpKT27N/J4KXdSHXB0Pj+XHfF7U5HAlTsIiIlkn70IInzOrIjzMMDNe/k1mv/z+lIBVTsIiLFlJOTzYCZ7dgYkUef6Ovp3eFfTkf6ExW7iEgxeNxuBk1rS1JkBneGNCbxjglORzqJiv043phad+rUqTzwwAMleu2oUaOoVasWjRs35tJLL+Wtt94q0muef/75Py3bsWMHDRs2PON6IlJ8D0/rzKrwA9zsrstjPWY4HadQKnY/M2jQINatW8eCBQvo168fubm5TkcSkWP+PaM7S1y/ck1ONZ7u9Z7TcU5JxX4GqampdO3alebNm9O8eXO++OILANasWUPLli1p0qQJLVu2ZMuWLSe9dtGiRbRo0YLk5GTi4+MLSvrw4cPUrVv3tKVdr149oqKiCuac2bZtG+3bt6dZs2ZcddVVbN682QfvVkROZfycAczxrKd5dhRjey0r00m9iss/r2Nf8jDs/sG72zz7MrjpmWK/LDExkUGDBtG6dWt+++032rVrx6ZNm6hfvz6rVq0iNDSUjz76iEceeYR58+YVvG7+/PmMHTuWxYsXU6VKFdq0acOiRYvo0qULs2fPpmvXroSFnXpioG+//ZZ69epRs2ZNAPr27cvkyZOpV68eq1ev5h//+AcrV64s/u9BRIrt9Q+eYGrGShrmhDPhnmWEh0c4Hem0/LPY/chHH33Ejz/+WPD88OHDpKenc+jQIXr16sXWrVsxxvxp9P3xxx+TlJTE8uXLqVixIgD3338/zz33HF26dOGNN97g1VdfLXR/48aN49VXX2X79u0sXZr/fSVHjhzhyy+/5I477ihYLzs7+5SZTzVFsKYOFim+eStf5uV9c4jPDWF81w+Ija7sdKQz8s9iL8HI2lc8Hg9fffUVFSpU+NPy/v37c+211zJ//nx27NhBmzZtCn52/vnns337dn766ScSEvKndWjVqhU7duzg008/xe12n3Ry8w+DBg1iyJAhvPvuu/Ts2ZNt27bh8XioXLky69atK1LmE6cNhvypg+Pj44vxzkXko9VzGfPrJGq4DWM7zOGsarWcjlQkOsZ+Bm3btuWll14qeP5HuR46dIhatfL/IU+dOvVPr6lTp05BMW/cuLFgec+ePenWrRt9+vQ5435vu+02EhISmDZtGhUrViQ+Pp65c+cCYK1l/fr1p3xtTEwM55xzDitWrADyS33p0qW0bt26aG9aRPhm4wqe2DCKKA88e/UU4mvVdzpSkanYj/PHFLx/3MaOHcvEiRNJSkqiUaNGXHrppQXT7w4dOpThw4fTqlUr3G73Sdu6+OKLmTVrFnfccQfbtm0D4J577iEtLY1u3boVKc+IESMYO3YsHo+HWbNm8dprr3H55ZfToEEDFixYULDe6NGj/5QbYPr06YwePZrGjRtz3XXXMXLkSC644ILS/opEyoUtv3zHo18l4jHwRNNnaXRRS6cjFYum7S1D77zzDgsWLGDGDP+59rU8/N5FiiNl7w7+vuAW9oR6GFnvITq07u10pAKattfP9O/fnyVLlrB48WKno4jIKaQdSmXQe11ICbc8dM69flXqxaFiLyMvvvii0xFE5DQyso4yYPZNbInI45+VbuHudg86HanEdIxdRMq9vLxcBk5vy7rIbLqHt6DfrU87HalUVOwiUq553G4emtqBryIO09lexNC7C/+MSSDxWrEbY1zGmO+MMQu9tU0REV8bOeMuPgr7netzz+aJHnOcjuMV3hyxJwKbvLg9ERGfeu7Nv/Ge2UyL7Io833uxX8//UhxeKXZjTBzQAfivN7bnFJfLRePGjQtuzzyT/wnYNm3acOLlmYVNz1vYeiLin16ZP5yZOV/ROCuC8T2XExp66rmbAo23rooZDwwFYr20PUdUqFChyB/bF5HA9eayF3jl0AdclONi4l1LiIqMdjqSV5V6xG6M6QjstdauPcN6fY0xScaYpNTU1NLuVkSkRBZ/Pp3xu17n3FzD+C4LqFKphtORvM4bI/ZWQCdjzM1AJFDRGDPTWtv9+JWstVOAKZD/ydPTbfDZNc+y+YB35xuvX7U+w64Ydtp1MjMzady4ccHz4cOHc+edd3o1h4g45/N1i3nqp2ep5DG8cMN04mrWdTqST5S62K21w4HhAMaYNsCQE0s9UBTnUIymxhUJLN9v/ZoRax8iBHiqxQQujm/idCSf8ctPnp5pZO0PTjU1bvXq1R1KJCKn8svOzQz79H4yQ+Dfl46geYPrnY7kU179gJK19hNrbUdvbtNf/fE1ebt37wYgKSmJ7Oxsateu7XAyETnenv07Gbzor6S6YEidf3LDlcF/eNUvR+xOOfEYe/v27QsueezQoUPBV9m1aNGCuXPnMmHCBG6++WY8Hg8xMTG89dZbhITow7wi/iL96EEGzruFX8I9DKj2V7pe/w+nI5UJFftxCptXHeCTTz4pdHnnzp3p3LmzDxOJSEnl5GSTOKsdGyJyua/Cddx7ywinI5UZDS9FJOh43G4GT2vHNxEZ3BHSiIF/neh0pDKlYheRoPPItFv5NHw/N+Wdx4ges5yOU+b8qtid+Dan8ky/bwlGo2f0ZJHrF67Oqcozvd93Oo4j/KbYIyMj2b9/v8qmjFhr2b9/P5GRkU5HEfGaiXMH8rbnOxKyohjXa3nQTOpVXH5z8jQuLo6UlBQ03UDZiYyMLPjya5FAN3XRv3n96Ec0yAljYvdlhIdHOB3JMX5T7GFhYcTHxzsdQ0QC0PyPJ/PS3repmxvChK4LiY2u7HQkR/nNoRgRkZJYueYdnvvlRWq4YexNb3FWtVpOR3Kc34zYRUSKK2njJzz+w0gigWevnsL5tRs4HckvqNhFJCD99Os6HvnqAfJC4JnGT9PoopZOR/IbOhQjIgEnZe8OHlzeg4Muy8MXPshVTTs5HcmvqNhFJKAcTN/HoPe6kBJmSTynF7dcfa/TkfyOil1EAkZmdgYD3rqJLeF59K3YgXvaD3U6kl9SsYtIQMjLy2XgtLZ8F5HF3eF/4e+3Pet0JL+lYhcRv+dxuxk6tSNfRhyik6ceD9/9mtOR/JqKXUT83qiZ3fgwbBfX55zFv3vOdTqO31Oxi4hfG/NWP+aziSuzY3m+z5JyO/9LcajYRcRvvfLeI8zM/oLLs8KZ0PNDQkPDnI4UEFTsIuKXZi8fxysH36dejosX71pKVGS005EChopdRPzOki9mMG7nfzk31zCu03yqVKrhdKSAoikFRMSvfLFuCU9ueYZYD4y5fiq1zz7f6UgBRyN2EfEbP2xdzb/WDiEEeOovE7jk/GZORwpIKnYR8Qu/7vqJYZ/cR0YIPHbJY1xx2Q1ORwpYpS52Y0ykMWaNMWa9MWajMeZxbwQTkfIjNW0Xgxbezp5QeLD232nbopvTkQKaN46xZwPXWWuPGGPCgM+NMUustV97YdsiEuTSjx4kcU5Htkd46F+1K3fc8IDTkQJeqYvd5n/79JFjT8OO3fSN1CJyRjk52Qyc2Z4fInPpE9mG+zrpD35v8MoxdmOMyxizDtgLfGitXe2N7YpI8PK43Qye1p41kUe53TRk8J0vOR0paHil2K21bmttYyAOuMIY0/DEdYwxfY0xScaYpNTUVG/sVkQC2KPTbuXT8H20z6vNyJ5vOR0nqHj1qhhr7UHgE6B9IT+bYq1NsNYm1KihDxuIlGdPzuzFQtcvXJVdhWd7f+B0nKDjjatiahhjKh97XAG4Adhc2u2KSHB6ce4gZru/pVlWBcb2WqpJvXzAG1fFnANMM8a4yP8fxRxr7UIvbFdEgsz0RU/x+tEPaZATxoR7lhIZEeV0pKDkjativgeaeCGLiASx9z6ZwsS9b3JenmH8be9TKaaq05GClj55KiI+9/E383hu+wSqu+GFdm9xdvXaTkcKapoETER86tsfP+Xx70cQATxz1WQuPO+ki+bEy1TsIuIzP/36PQ9/+U9yQuDpy5+i8cWtnY5ULuhQjIj4xK7UXxmy/B7SXPDwhYO5pllnpyOVGyp2EfG6g+n7GDi/M8lhlsSze9Dp6vucjlSuqNhFxKuysjMY8OZNbA7P4/6KN9H9pmFORyp3VOwi4jUet5uB09rxXWQWd4ddwT9vG+N0pHJJxS4iXmGt5aE3OvBFxEFu8VzIw/e87nSkckvFLiJeMWp6N5aH7eS6nJqM7vmO03HKNRW7iJTaC7P/zrts5C9ZMbzQR/O/OE3FLiKlMmXBY0zP+oxGWeFM6LGc0NAwpyOVeyp2ESmxtz8cz+S097gwx8XEOxcRHRXrdCRBxS4iJbT0y1mMTXmVc3IN4295h2qVz3Y6khyjKQVEpNi++n4pT25+ilgPPHfdG9Q+p57TkeQ4GrGLSLFs3PYNj33zIACj/zKOBhckOJxITqRiF5Ei+23XVh5a2YcjIfDYJY9y5WVtnY4khVCxi0iR7EvbzcCFt7MnFB6M60e7Fnc7HUlOQcUuImd0NCOdxDkd2Bbu5u/VbuOvNw5wOpKchopdRE4rNy+XxJlt+T4yh14Vrub+Tk84HUnOQMUuIqfkcbt58I12rI44wu00YPCdLzsdSYpAxS4ip/To9K58HJ5Ku9w4Rvaa7XQcKSIVu4gU6ulZvVkYso1W2ZV5rs9Cp+NIMajYReQkL80bwpt5a2maFcn4Xss0qVeAUbGLyJ/MWPwMr6Uv5ZJsFxPvWUZkRJTTkaSYNKWAiBRY8Ol/mbhnJuflGcbfuoBKMVWdjiQlUOoRuzGmtjHmY2PMJmPMRmNMojeCiUjZ+nTtAp7dNo6qbhjTdhbn1qjjdCQpIW+M2POAB6213xpjYoG1xpgPrbU/emHbIlIGvt38GSPXPUI48HTr/3BRnUZOR5JSKPWI3Vr7u7X222OP04FNQK3SbldEysbW3zYw/PO/k2Ng1OWjaVr/KqcjSSl59eSpMaYu0ARYXcjP+hpjkowxSampqd7crYiU0O/7khmyrBsHXJZh5w+kTcKtTkcSL/BasRtjYoB5wEBr7eETf26tnWKtTbDWJtSoUcNbuxWREjp05ACJ797Cb2GW/md1p3ObvzkdSbzEK8VujAkjv9RnWWvf9cY2RcR3srIzSJzVnk0Rbu6NaUvPm4c7HUm8yBtXxRjgNWCTtXZs6SOJiC953G4GTWvH2shM7nI1o//t+s822HhjxN4K6AFcZ4xZd+x2sxe2KyI+MHRqRz6POEhHz/k82n2q03HEB0p9uaO19nPAeCGLiPjY49PvYlloCtfm1ODJe3XUNFhpSgGRcmLs2//kHbuRK7JjGNtH878EMxW7SDnw3/dHMC3zUy7LDmNi9+WEhoY5HUl8SHPFiAS5OR9O5D/73+WC3BAm3rGI6KhYpyOJj2nELhLEln/1Fi+kvMLZeTCu4ztUr3KO05GkDGjELhKkvvp+Of/eNJoYC89e+xp1zr3I6UhSRjRiFwlCG7cl8a81g7DA6OYv0PDCvzgdScqQil0kyCT/vpWhK/uQ7oLH6j9Ci0btnY4kZUzFLhJE9h/czcAPbuf3MMvguL/RvuU9TkcSB6jYRYLE0Yx0BrzdgZ/D3fxflc7ceeNApyOJQ1TsIkEgLy+XxBlt+T4yhx6Rrenb+UmnI4mDVOwiAc7jdvPgG+1ZHXmE27iEIXdNdjqSOEzFLhLg/jX9dlaG76Vt7rmM7P6W03HED6jYRQLYM7Pu5f2Qn2mVXYkxfRZr/hcBVOwiAWvSuw/xZu4ammRFMr7XcpW6FFCxiwSgmUue5b+Hl1A/J5SJdy8hMiLK6UjiRzSlgEiAeX/Va0zYPYO4PMP4WxdQOba605HEz2jELhJAVn37Ps/+PJYqbnih7SzOrVHH6UjihzRiFwkQ67Z8zsjvhhMGPN1yEhfVaeR0JPFTKnaRALAteSMPf/Z/ZIfA6MueoNml1zgdSfyYDsWI+Lnd+5J5cEk39rlgaPwArmve1elI4udU7CJ+7NCRAwx8txO/hnvoX7MbXa7t53QkCQAqdhE/lZWdQeKs9myMyOPe6Bvp1eFRpyNJgFCxi/ghj9vN4GntWRuZyV2upvS/Y5zTkSSAqNhF/NCwqZ34LCKNDu54Hu0+zek4EmC8UuzGmNeNMXuNMRu8sT2R8uyJ6XezNPQ3rsmpzlO95jsdRwKQt0bsUwF9/5ZIKY17uz9z7Q9ckRXN2F5LNf+LlIhXit1auwo44I1tiZRXr3/wONMyP+ayrDDGd19KeHiE05EkQOkDSiJ+4J0Vk5i0by7n54Yw4a8LiY2u7HQkCWBldvLUGNPXGJNkjElKTU0tq92K+L0Pv36b5397mbPy4IUOc6hR5VynI0mAK7Nit9ZOsdYmWGsTatSoUVa7FfFrq3/4iH//+ARRHnimzX+Jr1Xf6UgSBHS5o4hDNm1fy6OrE/EATzQbQ6N6VzodSYKEty53fAv4CrjYGJNijLnPG9sVCVbJu7fz0IrepLvgkYuG0brxzU5HkiDilZOn1tpu3tiOSHmQdiiVQe/fyq5wy9Bz7+Pm1j2djiRBRodiRMpQRtZR+s9uz9ZwN/0qd+KutoOdjiRBSMUuUkby8nJJnH4j6yNz6B7Rin5dnnI6kgQpFbtIGfC43Qx54ya+jkjnVlufh7q94nQkCWIqdpEyMGL6X1kRvocbcs9hVI/ZTseRIKdiF/GxZ968nwUhP9EyuyJjei/S/C/icyp2ER/6z7vDeDPnaxpnRTCu53JCQ8OcjiTlgIpdxEdmLR3DlMOLqJ8Tyot3LyUqMtrpSFJOqNhFfOCDVa8z4fep1M41jO3yHpVjqzsdScoRze4o4mWffbeQZ35+gcoew/NtZxBXs67TkaScUbGLeNH6LV8w8tthhAJPtXiJi+o0djqSlEMqdhEv2Z68kYc/60dmCDzZ8HESGrRxOpKUUzrGLuIFe/bvZPDSbqS64KG6D3DdFbc7HUnKMRW7SCmlHz1I4ryO7Ajz8ECNO7ntur87HUnKORW7SCnk5GQzYGY7Nkbk0Sf6enp3/JfTkURU7CIl5XG7GTStLUmRGdwZ0pjEOyY4HUkEULGLlNjDUzuzKvwAN7vr8FiPGU7HESmgYhcpgX/P6M6S0F+5JrsaT/da4HQckT9RsYsU0/g5A5jjWU/z7CjG9l6mSb3E76jYRYrh9Q+eYGrGShpmhzLhnmWEh0c4HUnkJPqAkkgRzVvxMi/vm0N8bgjjuy4kNrqy05FECqURu0gRfLR6Ls//OokabhjbYQ5nVavldCSRU9KIXeQMvtm4gic2jKIC8OzVU4ivVd/pSCKnpRG7yGls+eU7Hv0qEY+BJ5o+S6OLWjodSeSMVOwip5C8+xce/KgnB12W4Rc9ROsmHZ2OJFIkXil2Y0x7Y8wWY8zPxpiHvbFNEScdOLSXge93YVeYZdC599GhdW+nI4kUWamL3RjjAiYBNwGXAt2MMZeWdrsiTsnIPEL/t2/m53A3/Sp1olu7wU5HEikWb4zYrwB+ttZut9bmALOBzl7YrkiZy83NYcCMtnwfkU2PiFb0u/UppyOJFJs3roqpBSQf9zwF+IsXtnuSscu38O53O32xaQkioTaXaDKJthlEHbuPJoNom0kUmUTZjEJ+nkmUPcqMqpmsrujiNs/FDOn2itNvRaREvFHsppBl9qSVjOkL9AU477zzSrSjOtWiuSK+aoleK37OWsJtJhHuTCI9R/Nv7gwiPUeJ8GSc8Pz4x5kFP4vwHCXSk0GYzTnj7jyEkBUSRZYrmuyQKLJCopgWG8ai6Bzah9RnVI+3y+BNi/iGN4o9Bah93PM4YNeJK1lrpwBTABISEk4q/qLo2iyOrs3iSvJS8RV3HuSkQ/aJt8OQfaSQZYWtm56/Des58/5cERARm3+LioWIihBxzv+WFdwqQkRMIcvyH4eERRFlDFHHNjt5/WTmrZtE13pdGdliJMYUNl4RCQzeKPZvgHrGmHhgJ3AXcLcXtiu+Yi3kZR0r3sLK9hQFXFiB52YUbZ/hJxZvLMSe9aeyLayA/7QsPAZCw738q7C8vP5lJq+fzC3n38K/rvyXSl0CXqmL3VqbZ4x5AFgGuIDXrbUbS51MTubxQM4pRsF/Wn6qkfFxyz15Z95fSOjJZRtdA6qef/Ly8JhTF3N4DIT430cmrLW8+N2LvPrDq3S5sAujWozCFaKZGiXweWVKAWvtYmCxN7YVlPJyjhVvUUfHhR3GODZiLoqwqJNHvJXrFDIKPs3oOCIGQiMhSEevbo+bp9c8zdtb3qZrva6MaDGCEON///MRKQnNFXMq1uYfZii0gI8UfWScnQ7u7CLs0JxcspGVoFLcKQo4pvBSDo8Fl/6xnk5WXhbDVg1jZfJK+jTow8BmA1XqElSCrwE87qKV7SkPYxy3vLgn8/4o4Iq1TnvirtBlYVFBOzr2J2lZaQxYOYD1qesZ1nwY3S/t7nQkEa8LrGL/cQHs+Pz0ZV3kk3knFm8sxNQ8/Ym7Qg9X6IsWAsXmA5tJXJnIvsx9jLlmDO3qtnM6kohPBFaxJ6+BH+YWfjKvoHiLcJWFn57ME99Z8ssSRnwxgooRFZl20zQaVm/odCQRnwmsYm/3ZP5NpIiy3dmMWzuOWZtm0bRmU15o8wLVK1R3OpaITwVWsYsUwy+HfmHoqqFsPrCZ7pd0Z3CzwYS5wpyOJeJzKnYJOh7rYe6Wubyw9gUiXBG8dN1LXFP7GqdjiZQZFbsEleT0ZEZ+OZJvdn/DledcyehWozkr+iynY4mUKRW7BIVcTy6zfpzFy+tfxmVcjGoxitvq3abpAaRcUrFLwPv69695evXTbD+0nWviruGxKx/j7OiznY4l4hgVuwSsrWlbefG7F/k4+WPiYuJ0LF3kGBW7BJyU9BReXvcyC7cvJDosmgFNBtCzQU8iXPqwmAio2CWA7Dyykzc2vMG8rfNwGRe9G/bmvob3USmiktPRRPyKil383oZ9G5i6cSof/vohIYRwa71b6deon652ETkFFbv4pVxPLquSVzFj0wzW7llLTFgMvRr04u76d+vEqMgZqNjFr/x6+FfmbZ3H+z+/z/6s/ZwdfTZDEobQtV5XYsJjnI4nEhBU7OK4A1kHWPHbChZtX8TaPWtxGRdXx11N13pdaVWrFaEh+tdUpDj0X4w4Ii0rjZW/rWTZjmWs2b0Gt3VTt2JdEpsm0vmCztSIquF0RJGApWKXMuGxHjYf2MxnKZ/x2c7P+GHfD3ish/Niz+PehvfSrm47LqpykT4pKuIFKnbxCWstyenJfLP7G77Z8w2rf1/Nvsx9ADSs1pB+jfrRpnYbLql6icpcxMtU7OIVWXlZbD6wmQ37NvD9vu9Zu2ctezP2AlAtshrNz27OVXFX0fLclpoPXcTHVOxSbEdzj7Lt4DZ+SvuJjfs3smHfBrambcVt3QDUjKpJs5rNSDg7gYSzE4h62fzlAAAFkElEQVSvGK9RuUgZUrFLoay1pGWnkZyezG+Hf+Pngz/n39J+ZtfRXQXrxYbHcln1y7i34b1cVv0yGlZvqBOfIg5TsZdjR3KOsDdjL7szdrPzyE6S05NJSU8hOT2Z5PRkjuYeLVg3NCSU8yudz+U1L+f2yrdzYeULubDyhcTFxmk0LuJnSlXsxpg7gFHAJcAV1tokb4SSkstx55CWlUZadhoHsg6QlpXGweyD7Mvcx96MvezJ2MOeo3vYm7GXjLyMP702LCSMWjG1qB1bm6Y1m1I7tnb+rWL+fViIvlZOJBCUdsS+AbgNeMULWco9j/WQ7c4mKy+LjLwMjuQcIT0nnSO5/7s/knOE9Nx0juT87/Hh7MP5JZ6d9qdR9vFCTSjVo6pzVtRZ1KtSj9a1WlMzqiZnRZ1Fzaia1IqpRc2omrhCXGX8rkXE20pV7NbaTUBA/ClurcVt3QX3Hus5+d7zv+du6ybXnUuu54TbicuOe57nyTtpWY47hyx3Fpl5mWTlZeXf3Pn3mXmZBY//WF4U4SHhxITHEBseS0xYDBXDKxIXG0fVyKpUiaySf4uoUvC4akRVKkZUJMSE+Pi3LCL+IKCOsU9eP5nFvyw+qYQLuz9xHYst06yhJpQwVxihIaFUcFUgMjSy4FbBVYFKUZX+t8wVSYXQCic9jw2PzS/wsPz7mLD8Mg93hZfpexGRwHLGYjfGfAQUNp3eo9baBUXdkTGmL9AX4LzzzitywOPVqFCDepXr4TIuQkJC8u/N/+6Pf1zYOsaYk15zuvXDXGGEhYQRHhJe8PhPtxOXHXseGhKq0bGIOMZYW/qRrDHmE2BIUU+eJiQk2KQknWcVESkOY8xaa23CmdbTsFJEJMiUqtiNMbcaY1KAFsAiY8wy78QSEZGSKu1VMfOB+V7KIiIiXqBDMSIiQUbFLiISZFTsIiJBRsUuIhJkVOwiIkHGKx9QKvZOjUkFfi3zHZdedWCf0yHKmN5z+aD3HBjqWGvP+IUHjhR7oDLGJBXlU1/BRO+5fNB7Di46FCMiEmRU7CIiQUbFXjxTnA7gAL3n8kHvOYjoGLuISJDRiF1EJMio2EvIGDPEGGONMdWdzuJrxpgxxpjNxpjvjTHzjTGVnc7kK8aY9saYLcaYn40xDzudx9eMMbWNMR8bYzYZYzYaYxKdzlRWjDEuY8x3xpiFTmfxNhV7CRhjagM3Ar85naWMfAg0tNY2An4ChjucxyeMMS5gEnATcCnQzRhzqbOpfC4PeNBaewlwJfDPcvCe/5AIbHI6hC+o2EtmHDAUyviLVB1irV1urc079vRrIM7JPD50BfCztXa7tTYHmA10djiTT1lrf7fWfnvscTr5RVfL2VS+Z4yJAzoA/3U6iy+o2IvJGNMJ2GmtXe90FofcCyxxOoSP1AKSj3ueQjkouT8YY+oCTYDVziYpE+PJH5x5nA7iC6X6oo1gdbov8AYeAdqWbSLfK8qXlhtjHiX/T/dZZZmtDJlClpWLv8qMMTHAPGCgtfaw03l8yRjTEdhrrV1rjGnjdB5fULEXwlp7Q2HLjTGXAfHAemMM5B+S+NYYc4W1dncZRvS6U73nPxhjegEdgett8F4jmwLUPu55HLDLoSxlxhgTRn6pz7LWvut0njLQCuhkjLkZiAQqGmNmWmu7O5zLa3QdeykYY3YACdbaQJtIqFiMMe2BscA11tpUp/P4ijEmlPyTw9cDO4FvgLuttRsdDeZDJn+EMg04YK0d6HSesnZsxD7EWtvR6SzepGPsUhQvAbHAh8aYdcaYyU4H8oVjJ4gfAJaRfxJxTjCX+jGtgB7Adcf+2a47NpKVAKYRu4hIkNGIXUQkyKjYRUSCjIpdRCTIqNhFRIKMil1EJMio2EVEgoyKXUQkyKjYRUSCzP8DxDHmZJY2zhoAAAAASUVORK5CYII=\n",
      "text/plain": [
       "<Figure size 432x288 with 1 Axes>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "import matplotlib.pyplot as plt\n",
    "import numpy as np\n",
    "\n",
    "def relu(x):\n",
    "    return max(0, x)\n",
    "\n",
    "def leaky(x, a):\n",
    "    if x <= 0:\n",
    "        return a*x\n",
    "    else:\n",
    "        return x\n",
    "\n",
    "def elu(x):\n",
    "    if x<= 0:\n",
    "        return np.exp(x) - 1\n",
    "    else:\n",
    "        return x\n",
    "\n",
    "X = np.arange(-5.0, 5.0, 0.05)\n",
    "R = np.array([relu(x) for x in X])\n",
    "L = np.array([leaky(x, 0.05) for x in X])\n",
    "E = np.array([elu(x) for x in X])\n",
    "\n",
    "plt.plot(X, R, label=\"ReLU\")\n",
    "plt.plot(X, L, label=\"Leaky ReLU\")\n",
    "plt.plot(X, E, label=\"ELU\")\n",
    "plt.legend()\n",
    "\n",
    "plt.show()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Recent progress in image recognition"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Other approaches to deep neural nets"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## On the future of neural networks"
   ]
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.6.5"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 2
}
